Sep 10 19:18:11.335422 ip-10-0-134-64 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 19:18:11.335435 ip-10-0-134-64 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 19:18:11.335446 ip-10-0-134-64 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 19:18:11.335862 ip-10-0-134-64 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 19:18:21.453134 ip-10-0-134-64 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 19:18:21.453153 ip-10-0-134-64 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5774659c0f3a487ba642cc566e1f8c0a -- Sep 10 19:20:13.920573 ip-10-0-134-64 systemd[1]: Starting Kubernetes Kubelet... Sep 10 19:20:14.304958 ip-10-0-134-64 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 19:20:14.304958 ip-10-0-134-64 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 19:20:14.304958 ip-10-0-134-64 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 19:20:14.304958 ip-10-0-134-64 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 19:20:14.304958 ip-10-0-134-64 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 19:20:14.306699 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.306608 2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 19:20:14.311513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311495 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:20:14.311513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311512 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311516 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311519 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311523 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311531 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311535 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311538 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311541 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311544 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311546 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311549 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311552 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311555 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311560 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311564 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311568 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311570 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311573 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311576 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311579 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:20:14.311582 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311581 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311584 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311587 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311590 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311593 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311596 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311598 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311601 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311604 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311607 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311609 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311612 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311614 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311617 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311620 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311622 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311625 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311668 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311713 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311718 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:20:14.312215 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311721 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311738 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311743 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311747 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311751 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311756 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311760 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311765 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311771 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311776 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311875 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311881 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311887 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311893 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311899 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311904 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311909 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311914 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311918 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311923 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:20:14.312999 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311927 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311938 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311942 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311946 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311951 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311958 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311964 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311968 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311973 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311977 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311982 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311987 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.311991 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312001 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312006 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312011 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312016 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312021 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312026 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:20:14.313522 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312030 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312035 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312041 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312047 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312054 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312061 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312071 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312076 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312081 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312086 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312090 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312095 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312099 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312103 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312107 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312112 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312117 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312122 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.312132 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313413 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:20:14.314033 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313422 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313427 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313432 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313435 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313438 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313441 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313444 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313447 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313449 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313452 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313454 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313457 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313461 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313464 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313468 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313471 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313475 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313478 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313480 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:20:14.314581 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313483 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313485 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313488 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313490 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313493 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313496 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313499 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313501 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313505 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313509 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313512 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313515 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313518 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313521 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313524 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313527 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313530 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313533 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313536 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313538 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:20:14.315065 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313541 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313552 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313555 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313558 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313561 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313563 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313566 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313568 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313571 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313573 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313576 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313579 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313582 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313584 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313587 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313591 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313593 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313596 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313599 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313601 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:20:14.315599 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313604 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313606 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313609 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313611 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313614 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313617 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313620 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313623 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313626 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313628 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313631 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313633 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313636 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313638 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313641 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313643 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313646 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313649 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313651 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313653 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:20:14.316110 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313656 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313659 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313662 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313664 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313666 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313669 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313672 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313674 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313678 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313682 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313685 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313687 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313690 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313693 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313695 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313698 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313701 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313703 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.313706 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314071 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:20:14.316626 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314076 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314079 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314082 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314084 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314088 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314090 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314093 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314095 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314099 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314103 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314106 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314108 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314111 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314115 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314118 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314120 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314123 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314125 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314128 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314130 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:20:14.317128 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314134 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314137 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314139 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314142 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314145 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314147 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314150 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314153 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314155 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314158 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314161 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314163 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314166 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314168 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314171 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314174 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314177 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314179 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314182 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:20:14.317643 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314185 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314187 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314190 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314192 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314195 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314197 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314200 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314202 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314205 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314207 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314210 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314212 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314215 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314218 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314221 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314223 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314226 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314228 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314231 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314233 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:20:14.318113 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314236 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314239 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314241 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314244 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314246 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314249 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314252 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314254 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314257 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314260 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314262 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314265 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314268 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314270 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314272 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314288 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314291 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314294 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314296 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314299 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:20:14.318614 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314302 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314304 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314307 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314312 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314315 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314318 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314323 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314326 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314329 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314332 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314335 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314338 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314341 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314344 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314347 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314351 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314354 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314357 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.314360 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314436 2584 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 19:20:14.319137 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314476 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314487 2584 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314491 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314495 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314498 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314503 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314512 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314515 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314518 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314521 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314525 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314527 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314530 2584 flags.go:64] FLAG: --cgroup-root="" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314533 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314536 2584 flags.go:64] FLAG: --client-ca-file="" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314539 2584 flags.go:64] FLAG: --cloud-provider="external" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314542 2584 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314546 2584 flags.go:64] FLAG: --cluster-domain="" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314552 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314555 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314559 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314562 2584 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314566 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 19:20:14.319695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314570 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314573 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314577 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314580 2584 flags.go:64] FLAG: --contention-profiling="false" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314583 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314586 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314590 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314593 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314597 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314600 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314602 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314605 2584 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314608 2584 flags.go:64] FLAG: --enable-server="true" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314611 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314616 2584 flags.go:64] FLAG: --event-burst="100" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314619 2584 flags.go:64] FLAG: --event-qps="50" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314622 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314625 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314627 2584 flags.go:64] FLAG: --eviction-hard="" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314631 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314634 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314637 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314640 2584 flags.go:64] FLAG: --eviction-soft="" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314642 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314645 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 19:20:14.320258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314648 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314651 2584 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314668 2584 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314672 2584 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314676 2584 flags.go:64] FLAG: --feature-gates="" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314681 2584 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314684 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314689 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314692 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314695 2584 flags.go:64] FLAG: --healthz-port="10248" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314699 2584 flags.go:64] FLAG: --help="false" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314702 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314705 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314708 2584 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314711 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314714 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314718 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314721 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314724 2584 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314726 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314729 2584 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314732 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314736 2584 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314738 2584 flags.go:64] FLAG: --kube-reserved="" Sep 10 19:20:14.320896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314741 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314744 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314747 2584 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314750 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314753 2584 flags.go:64] FLAG: --lock-file="" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314755 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314758 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314761 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314766 2584 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314769 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314773 2584 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314775 2584 flags.go:64] FLAG: --logging-format="text" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314778 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314781 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314785 2584 flags.go:64] FLAG: --manifest-url="" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314789 2584 flags.go:64] FLAG: --manifest-url-header="" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314793 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314796 2584 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314800 2584 flags.go:64] FLAG: --max-pods="110" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314803 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314806 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314808 2584 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314811 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314814 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314817 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 19:20:14.321521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314820 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314827 2584 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314830 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314833 2584 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314836 2584 flags.go:64] FLAG: --pod-cidr="" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314839 2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314844 2584 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314846 2584 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314850 2584 flags.go:64] FLAG: --pods-per-core="0" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314852 2584 flags.go:64] FLAG: --port="10250" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314855 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314858 2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dba94f1d5c558016" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314861 2584 flags.go:64] FLAG: --qos-reserved="" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314864 2584 flags.go:64] FLAG: --read-only-port="10255" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314867 2584 flags.go:64] FLAG: --register-node="true" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314869 2584 flags.go:64] FLAG: --register-with-taints="" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314875 2584 flags.go:64] FLAG: --registry-burst="10" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314880 2584 flags.go:64] FLAG: --registry-qps="5" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314882 2584 flags.go:64] FLAG: --reserved-cpus="" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314885 2584 flags.go:64] FLAG: --reserved-memory="" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314889 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314892 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314895 2584 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314900 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314902 2584 flags.go:64] FLAG: --runonce="false" Sep 10 19:20:14.322237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314905 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314908 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314911 2584 flags.go:64] FLAG: --seccomp-default="false" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314914 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314917 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314920 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314923 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314926 2584 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314929 2584 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314932 2584 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314934 2584 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314937 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314940 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314943 2584 flags.go:64] FLAG: --system-cgroups="" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314946 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314951 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314954 2584 flags.go:64] FLAG: --tls-cert-file="" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314957 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314961 2584 flags.go:64] FLAG: --tls-min-version="" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314963 2584 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314966 2584 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314969 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314972 2584 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314974 2584 flags.go:64] FLAG: --v="2" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314980 2584 flags.go:64] FLAG: --version="false" Sep 10 19:20:14.322911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314983 2584 flags.go:64] FLAG: --vmodule="" Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314987 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.314990 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315099 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315103 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315109 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315112 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315115 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315118 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315120 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315123 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315126 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315128 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315132 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315134 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315137 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315139 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315143 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315147 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315150 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:20:14.323589 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315153 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315156 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315158 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315161 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315163 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315166 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315169 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315171 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315174 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315176 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315179 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315183 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315186 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315188 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315191 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315193 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315196 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315200 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315203 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315207 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:20:14.324083 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315210 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315214 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315217 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315220 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315222 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315225 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315227 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315230 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315233 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315236 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315238 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315241 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315243 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315246 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315248 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315251 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315254 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315256 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315259 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315261 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:20:14.324605 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315264 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315266 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315269 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315273 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315311 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315314 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315317 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315319 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315322 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315326 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315329 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315332 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315335 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315338 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315340 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315343 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315345 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315348 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315351 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315353 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:20:14.325093 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315356 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315358 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315361 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315363 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315366 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315368 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315371 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315373 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315376 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315379 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315381 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315384 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315386 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315389 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315392 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315394 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315397 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315400 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315403 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:20:14.325624 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315405 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315409 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.315414 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.315904 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.323000 2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.323020 2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.323040 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.323046 2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323100 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323105 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323109 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323111 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323115 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323118 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:20:14.326129 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323121 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323124 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323127 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323129 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323132 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323135 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323137 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323140 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323143 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323145 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323148 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323151 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323154 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323156 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323159 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323162 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323164 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323167 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323169 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323172 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:20:14.326510 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323174 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323177 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323179 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323184 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323187 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323190 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323192 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323195 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323198 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323201 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323203 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323206 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323209 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323212 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323214 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323217 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323220 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323222 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323224 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323227 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:20:14.327004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323230 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323232 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323235 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323238 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323240 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323243 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323245 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323248 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323251 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323253 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323256 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323258 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323261 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323264 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323266 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323269 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323272 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323298 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323302 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323304 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:20:14.327513 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323307 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323311 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323315 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323318 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323321 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323324 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323327 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323330 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323332 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323335 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323338 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323341 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323343 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323346 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323349 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323351 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323354 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323356 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323359 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323361 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:20:14.328007 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323364 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323367 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323371 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323375 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323379 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323382 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323384 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323387 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323391 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323394 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323397 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323400 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323402 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.323407 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 19:20:14.328521 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323520 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323525 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323528 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323530 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323533 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323535 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323538 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323540 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323543 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323546 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323548 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323551 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323553 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323556 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323559 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323561 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323564 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323566 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323568 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323571 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:20:14.328867 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323573 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323576 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323578 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323581 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323583 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323586 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323588 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323591 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323594 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323597 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323600 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323603 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323605 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323608 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323611 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323613 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323616 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323619 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323621 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323624 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:20:14.329432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323626 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323630 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323634 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323636 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323639 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323642 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323644 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323647 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323649 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323652 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323654 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323657 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323659 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323662 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323664 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323667 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323669 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323672 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323675 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:20:14.329929 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323678 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323680 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323683 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323686 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323688 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323691 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323694 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323697 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323699 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323702 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323705 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323707 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323709 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323712 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323715 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323717 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323720 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323722 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323725 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:20:14.330417 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323729 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323732 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323735 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323738 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323740 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323744 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323748 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323751 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323754 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323757 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323760 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323763 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323765 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323768 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323771 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323773 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323777 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323779 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323781 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:20:14.330877 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323784 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:20:14.331382 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:14.323787 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:20:14.331382 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.323792 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 19:20:14.331382 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.323889 2584 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 19:20:14.331382 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.325842 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 19:20:14.331382 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.326666 2584 server.go:1019] "Starting client certificate rotation" Sep 10 19:20:14.331382 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.326767 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 19:20:14.331382 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.327505 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 19:20:14.331382 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.329161 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 19:20:14.332347 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.331888 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 19:20:14.340825 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.340805 2584 log.go:25] "Validated CRI v1 runtime API" Sep 10 19:20:14.344910 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.344892 2584 log.go:25] "Validated CRI v1 image API" Sep 10 19:20:14.345760 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.345743 2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 19:20:14.349201 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.349179 2584 fs.go:135] Filesystem UUIDs: map[1edc0a86-a63d-4439-b8b7-13821e9d1f72:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 eafe1f16-0731-412a-ae13-9bd677a446a1:/dev/nvme0n1p4] Sep 10 19:20:14.349290 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.349198 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 19:20:14.354136 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.354021 2584 manager.go:216] Machine: {Timestamp:2026-09-10 19:20:14.352815101 +0000 UTC m=+0.320627339 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090028 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26241a2593351643a14008ee491537 SystemUUID:ec26241a-2593-3516-43a1-4008ee491537 BootID:5774659c-0f3a-487b-a642-cc566e1f8c0a Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:18:1c:23:66:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:18:1c:23:66:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:cd:1b:6a:b3:f7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 19:20:14.354136 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.354132 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 19:20:14.354248 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.354205 2584 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 19:20:14.354505 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.354489 2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 19:20:14.357107 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.357080 2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 19:20:14.357241 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.357111 2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-64.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 19:20:14.357797 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.357786 2584 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 19:20:14.357830 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.357799 2584 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 19:20:14.357830 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.357806 2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 19:20:14.358885 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.358875 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 19:20:14.360129 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.360120 2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 19:20:14.363889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.363877 2584 state_mem.go:36] "Initialized new in-memory state store" Sep 10 19:20:14.364002 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.363992 2584 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 19:20:14.364034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.364028 2584 kubelet.go:486] "Attempting to sync node with API server" Sep 10 19:20:14.364061 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.364042 2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 19:20:14.364061 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.364057 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 19:20:14.364121 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.364066 2584 kubelet.go:398] "Adding apiserver pod source" Sep 10 19:20:14.364121 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.364075 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 19:20:14.364450 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.364436 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 19:20:14.364509 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.364454 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 19:20:14.366947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.366930 2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 19:20:14.369513 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.369499 2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 19:20:14.369575 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.369528 2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 19:20:14.369695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.369685 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 19:20:14.369743 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.369703 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 19:20:14.369743 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.369713 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 19:20:14.369743 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.369721 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 19:20:14.369743 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.369732 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 19:20:14.369743 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.369742 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 19:20:14.370185 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.370172 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 19:20:14.370217 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.370187 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 19:20:14.370217 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.370197 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 19:20:14.370217 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.370205 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 19:20:14.370346 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.370227 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 19:20:14.370346 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.370239 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 19:20:14.370962 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.370951 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 19:20:14.370994 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.370964 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 19:20:14.372515 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.372493 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 19:20:14.372589 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.372494 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 19:20:14.374511 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.374498 2584 server.go:1268] "Started kubelet" Sep 10 19:20:14.374572 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.374549 2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 19:20:14.374702 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.374646 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 19:20:14.374802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.374722 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 19:20:14.375534 ip-10-0-134-64 systemd[1]: Started Kubernetes Kubelet. Sep 10 19:20:14.378034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.377936 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9vzc2" Sep 10 19:20:14.378173 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.378154 2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 19:20:14.379076 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.379056 2584 server.go:310] "Adding debug handlers to kubelet server" Sep 10 19:20:14.380942 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.380914 2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 19:20:14.381205 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.381186 2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 19:20:14.381205 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.381200 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 19:20:14.381436 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.381404 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:14.381741 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.381722 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 19:20:14.381741 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.381725 2584 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 19:20:14.381868 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.381748 2584 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 19:20:14.381868 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.381836 2584 reconstruct.go:97] "Volume reconstruction finished" Sep 10 19:20:14.381868 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.381844 2584 reconciler.go:29] "Reconciler: start to sync state" Sep 10 19:20:14.382128 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.382112 2584 factory.go:55] Registering systemd factory Sep 10 19:20:14.382206 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.382131 2584 factory.go:223] Registration of the systemd container factory successfully Sep 10 19:20:14.382453 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.382430 2584 factory.go:153] Registering CRI-O factory Sep 10 19:20:14.382453 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.382449 2584 factory.go:223] Registration of the crio container factory successfully Sep 10 19:20:14.382619 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.382500 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 19:20:14.382619 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.382520 2584 factory.go:103] Registering Raw factory Sep 10 19:20:14.382619 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.382536 2584 manager.go:1196] Started watching for new ooms in manager Sep 10 19:20:14.382619 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.382541 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-64.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 19:20:14.382856 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.382840 2584 manager.go:349] Starting recovery of all containers Sep 10 19:20:14.383577 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.382606 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-64.ec2.internal.18d40be153cdd114 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-64.ec2.internal,UID:ip-10-0-134-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-64.ec2.internal,},FirstTimestamp:2026-09-10 19:20:14.374474004 +0000 UTC m=+0.342286241,LastTimestamp:2026-09-10 19:20:14.374474004 +0000 UTC m=+0.342286241,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-64.ec2.internal,}" Sep 10 19:20:14.385631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.385610 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9vzc2" Sep 10 19:20:14.387556 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.387529 2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 19:20:14.387666 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.387644 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 19:20:14.392788 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.392651 2584 manager.go:354] Recovery completed Sep 10 19:20:14.397113 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.397096 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:20:14.400053 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.399980 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:20:14.400053 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.400012 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:20:14.400053 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.400027 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:20:14.400708 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.400694 2584 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 19:20:14.400708 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.400705 2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 19:20:14.400839 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.400720 2584 state_mem.go:36] "Initialized new in-memory state store" Sep 10 19:20:14.400839 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.400764 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:20:14.401478 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.401462 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:20:14.401556 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.401487 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:20:14.401556 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.401496 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:20:14.401901 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.401886 2584 policy_none.go:49] "None policy: Start" Sep 10 19:20:14.401901 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.401903 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 19:20:14.401993 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.401912 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 19:20:14.403056 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.403046 2584 policy_none.go:47] "Start" Sep 10 19:20:14.425910 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.425894 2584 manager.go:337] "Starting Device Plugin manager" Sep 10 19:20:14.443862 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.425934 2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 19:20:14.443862 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.425947 2584 server.go:87] "Starting device plugin registration server" Sep 10 19:20:14.443862 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.426274 2584 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 19:20:14.443862 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.426305 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 19:20:14.443862 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.426651 2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 19:20:14.443862 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.426713 2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 19:20:14.443862 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.426720 2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 19:20:14.443862 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.427451 2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 19:20:14.443862 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.427508 2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:14.480321 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.480293 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 19:20:14.481564 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.481533 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 19:20:14.481659 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.481590 2584 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 19:20:14.481659 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.481620 2584 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 19:20:14.481769 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.481712 2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 19:20:14.483967 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.483948 2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:20:14.526853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.526837 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:20:14.529419 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.529401 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:20:14.529490 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.529437 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:20:14.529490 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.529449 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:20:14.529490 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.529471 2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.537078 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.537063 2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.537158 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.537089 2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-64.ec2.internal\": node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:14.546505 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.546482 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:14.582036 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.581965 2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal"] Sep 10 19:20:14.582036 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.582033 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:20:14.583131 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.583115 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:20:14.583203 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.583148 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:20:14.583203 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.583160 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:20:14.584578 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.584566 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:20:14.584704 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.584689 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.584739 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.584717 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:20:14.585334 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.585317 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:20:14.585428 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.585317 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:20:14.585428 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.585380 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:20:14.585428 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.585398 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:20:14.585572 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.585347 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:20:14.585572 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.585477 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:20:14.586802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.586788 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.586856 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.586815 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:20:14.587582 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.587566 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:20:14.587652 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.587596 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:20:14.587652 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.587610 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:20:14.617882 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.617860 2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-64.ec2.internal\" not found" node="ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.622071 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.622056 2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-64.ec2.internal\" not found" node="ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.646909 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.646889 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:14.682766 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.682743 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0af987b696cdc76ac0230a918c83b621-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal\" (UID: \"0af987b696cdc76ac0230a918c83b621\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.682854 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.682767 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0af987b696cdc76ac0230a918c83b621-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal\" (UID: \"0af987b696cdc76ac0230a918c83b621\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.682854 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.682784 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a45295e4379262bb71d3ff91072070b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-64.ec2.internal\" (UID: \"a45295e4379262bb71d3ff91072070b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.747909 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.747877 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:14.783226 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.783200 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0af987b696cdc76ac0230a918c83b621-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal\" (UID: \"0af987b696cdc76ac0230a918c83b621\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.783339 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.783226 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0af987b696cdc76ac0230a918c83b621-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal\" (UID: \"0af987b696cdc76ac0230a918c83b621\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.783339 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.783269 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0af987b696cdc76ac0230a918c83b621-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal\" (UID: \"0af987b696cdc76ac0230a918c83b621\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.783339 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.783305 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a45295e4379262bb71d3ff91072070b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-64.ec2.internal\" (UID: \"a45295e4379262bb71d3ff91072070b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.783467 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.783341 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a45295e4379262bb71d3ff91072070b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-64.ec2.internal\" (UID: \"a45295e4379262bb71d3ff91072070b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.783467 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.783370 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0af987b696cdc76ac0230a918c83b621-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal\" (UID: \"0af987b696cdc76ac0230a918c83b621\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.848495 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.848442 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:14.921046 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.921022 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.925455 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:14.924934 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal" Sep 10 19:20:14.948907 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:14.948885 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:15.052240 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:15.052202 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:15.156096 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:15.156006 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:15.261458 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:15.261431 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:15.328812 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.328792 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 19:20:15.329241 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.328952 2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 19:20:15.362583 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:15.362558 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-64.ec2.internal\" not found" Sep 10 19:20:15.387465 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.387431 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 19:15:14 +0000 UTC" deadline="2028-06-13 14:29:55.042333166 +0000 UTC" Sep 10 19:20:15.387465 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.387459 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15403h9m39.654878521s" Sep 10 19:20:15.402951 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.402933 2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:20:15.411973 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.411913 2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:20:15.482345 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.481753 2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" Sep 10 19:20:15.497026 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.497005 2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 19:20:15.499032 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.499013 2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal" Sep 10 19:20:15.505784 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.505763 2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 19:20:15.574766 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:15.574739 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af987b696cdc76ac0230a918c83b621.slice/crio-1f4df960f4d1a7d21b2fda9f96537cd88c9146c7ebf1b961042dd0570acf4930 WatchSource:0}: Error finding container 1f4df960f4d1a7d21b2fda9f96537cd88c9146c7ebf1b961042dd0570acf4930: Status 404 returned error can't find the container with id 1f4df960f4d1a7d21b2fda9f96537cd88c9146c7ebf1b961042dd0570acf4930 Sep 10 19:20:15.575129 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:15.575105 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45295e4379262bb71d3ff91072070b1.slice/crio-1c5ffff53f963dc51775de5d51e75b2f82780411b12b66eb1d88a049de54f143 WatchSource:0}: Error finding container 1c5ffff53f963dc51775de5d51e75b2f82780411b12b66eb1d88a049de54f143: Status 404 returned error can't find the container with id 1c5ffff53f963dc51775de5d51e75b2f82780411b12b66eb1d88a049de54f143 Sep 10 19:20:15.579850 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.579834 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:20:15.793537 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:15.793449 2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:20:16.166830 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.166738 2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:20:16.365047 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.365007 2584 apiserver.go:52] "Watching apiserver" Sep 10 19:20:16.373869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.373841 2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 19:20:16.374389 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.374362 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-f89b2","openshift-dns/node-resolver-jfqm6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal","openshift-multus/multus-additional-cni-plugins-m4tb5","openshift-multus/multus-vmbsk","openshift-multus/network-metrics-daemon-4fqpp","openshift-network-diagnostics/network-check-target-gwpjk","kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls","openshift-cluster-node-tuning-operator/tuned-d9bxz","openshift-image-registry/node-ca-jzlhg","openshift-network-operator/iptables-alerter-qqjvc","openshift-ovn-kubernetes/ovnkube-node-g9q29","kube-system/global-pull-secret-syncer-jclqb"] Sep 10 19:20:16.376309 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.376101 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:16.376309 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.376186 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:16.377681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.377658 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.379073 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.379048 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:16.379177 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.379125 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:16.380609 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.380588 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 19:20:16.380710 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.380693 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 19:20:16.380772 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.380587 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xmln5\"" Sep 10 19:20:16.380879 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.380858 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 19:20:16.381075 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.381054 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 19:20:16.381454 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.381230 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 19:20:16.381454 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.381246 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.381894 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.381877 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 19:20:16.382625 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.382593 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.382701 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.382593 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 19:20:16.383810 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.383790 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g2g28\"" Sep 10 19:20:16.383910 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.383853 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 19:20:16.384080 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.384062 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 19:20:16.384153 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.384124 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 19:20:16.384263 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.384247 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 19:20:16.384360 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.384329 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 19:20:16.385072 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.384788 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-th4lz\"" Sep 10 19:20:16.385072 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.385055 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 19:20:16.385519 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.385497 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:16.387770 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.387752 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 19:20:16.388054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.388035 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-46q5h\"" Sep 10 19:20:16.388140 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.388042 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 19:20:16.389204 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.388524 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.391452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.391098 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rdwrv\"" Sep 10 19:20:16.391452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.391143 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 19:20:16.391452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.391167 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.391452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.391404 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 19:20:16.391452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.391415 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 19:20:16.392976 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.392950 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-systemd-units\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.393075 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.392988 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-var-lib-openvswitch\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.393075 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393037 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-run-ovn\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.393075 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393063 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31af1687-310d-4c90-9c64-63c8639315f4-ovnkube-config\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.393238 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393087 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.393238 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393113 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.393238 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393147 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-system-cni-dir\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.393238 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393170 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-cnibin\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.393238 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393195 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-run-netns\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.393238 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393218 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-hostroot\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393241 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-conf-dir\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393264 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-run-multus-certs\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393339 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393382 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31af1687-310d-4c90-9c64-63c8639315f4-env-overrides\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393407 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqpv\" (UniqueName: \"kubernetes.io/projected/31af1687-310d-4c90-9c64-63c8639315f4-kube-api-access-ptqpv\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393446 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crqhh\" (UniqueName: \"kubernetes.io/projected/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-kube-api-access-crqhh\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393472 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-run-k8s-cni-cncf-io\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393496 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-kubelet\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393526 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-slash\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393536 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-79rnn\"" Sep 10 19:20:16.393577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393541 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-log-socket\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393610 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-socket-dir-parent\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393635 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-var-lib-cni-bin\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393666 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-etc-kubernetes\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393705 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31af1687-310d-4c90-9c64-63c8639315f4-ovn-node-metrics-cert\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393754 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-var-lib-cni-multus\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393780 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393783 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-var-lib-kubelet\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393835 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393833 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-run-openvswitch\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393961 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-etc-openvswitch\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393979 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.393996 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-system-cni-dir\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394011 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-cnibin\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394031 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394083 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-cni-dir\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.394132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394120 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwblm\" (UniqueName: \"kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm\") pod \"network-check-target-gwpjk\" (UID: \"71bc58ba-c381-420f-aa47-c298bda76828\") " pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394150 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394176 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8337463-d34f-4d10-9ed5-7819cf4c247a-cni-binary-copy\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394203 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-daemon-config\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394233 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdwv\" (UniqueName: \"kubernetes.io/projected/a8337463-d34f-4d10-9ed5-7819cf4c247a-kube-api-access-4zdwv\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394279 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-cni-bin\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394313 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-cni-netd\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394339 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31af1687-310d-4c90-9c64-63c8639315f4-ovnkube-script-lib\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394393 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394419 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn965\" (UniqueName: \"kubernetes.io/projected/29e94806-d52b-476f-bf6b-11fab94483a6-kube-api-access-tn965\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394443 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-run-netns\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394468 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-run-systemd\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394490 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-node-log\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.394947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.394670 2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 19:20:16.396241 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.395900 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.398906 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.398362 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 19:20:16.398906 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.398384 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 19:20:16.398906 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.398433 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.398906 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.398551 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 19:20:16.398906 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.398803 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nv5lh\"" Sep 10 19:20:16.403485 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.403466 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:16.403597 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.403573 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:16.403687 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.403662 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:16.405035 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.404795 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:20:16.405035 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.404920 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 19:20:16.405035 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.404970 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fqwr5\"" Sep 10 19:20:16.405244 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.405120 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 19:20:16.405873 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.405850 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 19:20:16.406775 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.406318 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-phmnk\"" Sep 10 19:20:16.406775 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.406532 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 19:20:16.419112 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.419007 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rmklc" Sep 10 19:20:16.427903 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.427876 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rmklc" Sep 10 19:20:16.482692 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.482665 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 19:20:16.488997 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.488945 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" event={"ID":"0af987b696cdc76ac0230a918c83b621","Type":"ContainerStarted","Data":"1f4df960f4d1a7d21b2fda9f96537cd88c9146c7ebf1b961042dd0570acf4930"} Sep 10 19:20:16.491334 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.491293 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal" event={"ID":"a45295e4379262bb71d3ff91072070b1","Type":"ContainerStarted","Data":"1c5ffff53f963dc51775de5d51e75b2f82780411b12b66eb1d88a049de54f143"} Sep 10 19:20:16.495116 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495090 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.495226 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495127 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31af1687-310d-4c90-9c64-63c8639315f4-env-overrides\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.495226 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495154 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqpv\" (UniqueName: \"kubernetes.io/projected/31af1687-310d-4c90-9c64-63c8639315f4-kube-api-access-ptqpv\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.495226 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495181 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8866770d-bbc0-47e6-8007-06762e2c140f-tmp\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.495226 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495206 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vzjh\" (UniqueName: \"kubernetes.io/projected/2ed20544-9b54-4e94-9ebb-7cf426aabff9-kube-api-access-7vzjh\") pod \"node-ca-jzlhg\" (UID: \"2ed20544-9b54-4e94-9ebb-7cf426aabff9\") " pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.495226 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495212 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495229 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-dbus\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495255 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495283 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-kubelet\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495329 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-log-socket\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495357 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqc7c\" (UniqueName: \"kubernetes.io/projected/1a11e036-f7b7-4856-ad1a-96a0872133e5-kube-api-access-sqc7c\") pod \"node-resolver-jfqm6\" (UID: \"1a11e036-f7b7-4856-ad1a-96a0872133e5\") " pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495384 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ed20544-9b54-4e94-9ebb-7cf426aabff9-serviceca\") pod \"node-ca-jzlhg\" (UID: \"2ed20544-9b54-4e94-9ebb-7cf426aabff9\") " pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495411 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-var-lib-kubelet\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495419 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-kubelet\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495454 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d45fd229-64fe-4f71-a53f-d20d4591c8df-agent-certs\") pod \"konnectivity-agent-f89b2\" (UID: \"d45fd229-64fe-4f71-a53f-d20d4591c8df\") " pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495477 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-sys\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495491 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-log-socket\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.495514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495501 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-var-lib-kubelet\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495503 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495547 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-socket-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495598 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ed20544-9b54-4e94-9ebb-7cf426aabff9-host\") pod \"node-ca-jzlhg\" (UID: \"2ed20544-9b54-4e94-9ebb-7cf426aabff9\") " pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495629 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-run-openvswitch\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495660 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495680 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-run-openvswitch\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495687 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-system-cni-dir\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495714 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495715 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-cnibin\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495743 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-cni-dir\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495759 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8337463-d34f-4d10-9ed5-7819cf4c247a-cni-binary-copy\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495760 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-cnibin\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495777 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a11e036-f7b7-4856-ad1a-96a0872133e5-tmp-dir\") pod \"node-resolver-jfqm6\" (UID: \"1a11e036-f7b7-4856-ad1a-96a0872133e5\") " pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495793 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2232257f-864c-4500-b080-7bcdf2a062f1-iptables-alerter-script\") pod \"iptables-alerter-qqjvc\" (UID: \"2232257f-864c-4500-b080-7bcdf2a062f1\") " pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495812 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31af1687-310d-4c90-9c64-63c8639315f4-ovnkube-script-lib\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495804 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-system-cni-dir\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.496211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495830 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495848 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn965\" (UniqueName: \"kubernetes.io/projected/29e94806-d52b-476f-bf6b-11fab94483a6-kube-api-access-tn965\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495833 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-cni-dir\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.495900 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-run-netns\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496101 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-node-log\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496140 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-sysconfig\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496168 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496197 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-systemd-units\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496225 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31af1687-310d-4c90-9c64-63c8639315f4-ovnkube-config\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496250 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.496278 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496321 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-run-netns\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496368 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-systemd-units\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.496381 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs podName:29e94806-d52b-476f-bf6b-11fab94483a6 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:16.99635202 +0000 UTC m=+2.964164268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs") pod "network-metrics-daemon-4fqpp" (UID: "29e94806-d52b-476f-bf6b-11fab94483a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496400 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-node-log\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496426 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-system-cni-dir\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496445 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-lib-modules\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.497082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496483 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-sys-fs\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496486 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8337463-d34f-4d10-9ed5-7819cf4c247a-cni-binary-copy\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496500 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-cnibin\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496515 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-run-netns\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496535 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-run-multus-certs\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496580 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31af1687-310d-4c90-9c64-63c8639315f4-ovnkube-script-lib\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496593 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-kubernetes\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496597 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-run-multus-certs\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496637 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-run\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496647 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-cnibin\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496654 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-system-cni-dir\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496647 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-run-netns\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496679 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-var-lib-kubelet\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496754 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8866770d-bbc0-47e6-8007-06762e2c140f-etc-tuned\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496786 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km7bq\" (UniqueName: \"kubernetes.io/projected/2232257f-864c-4500-b080-7bcdf2a062f1-kube-api-access-km7bq\") pod \"iptables-alerter-qqjvc\" (UID: \"2232257f-864c-4500-b080-7bcdf2a062f1\") " pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496820 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-slash\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496847 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-socket-dir-parent\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496860 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-slash\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.497899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496872 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-var-lib-cni-bin\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496884 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496891 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31af1687-310d-4c90-9c64-63c8639315f4-env-overrides\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496898 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-etc-kubernetes\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496929 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-socket-dir-parent\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496941 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-etc-kubernetes\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496946 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-var-lib-cni-bin\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.496970 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkr2\" (UniqueName: \"kubernetes.io/projected/8866770d-bbc0-47e6-8007-06762e2c140f-kube-api-access-nzkr2\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497002 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31af1687-310d-4c90-9c64-63c8639315f4-ovn-node-metrics-cert\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497027 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-var-lib-cni-multus\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497052 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-modprobe-d\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497074 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-host\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497080 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-var-lib-cni-multus\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497100 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tksk\" (UniqueName: \"kubernetes.io/projected/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-kube-api-access-2tksk\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497151 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-etc-openvswitch\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497105 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31af1687-310d-4c90-9c64-63c8639315f4-ovnkube-config\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497180 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.498631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497208 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-etc-openvswitch\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497209 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-daemon-config\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497234 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdwv\" (UniqueName: \"kubernetes.io/projected/a8337463-d34f-4d10-9ed5-7819cf4c247a-kube-api-access-4zdwv\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497261 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d45fd229-64fe-4f71-a53f-d20d4591c8df-konnectivity-ca\") pod \"konnectivity-agent-f89b2\" (UID: \"d45fd229-64fe-4f71-a53f-d20d4591c8df\") " pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497286 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-sysctl-d\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497330 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwblm\" (UniqueName: \"kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm\") pod \"network-check-target-gwpjk\" (UID: \"71bc58ba-c381-420f-aa47-c298bda76828\") " pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497358 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497425 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-sysctl-conf\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497451 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-systemd\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497477 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-cni-bin\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497504 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-cni-netd\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497531 2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497627 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497669 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-run-systemd\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497692 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497710 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crqhh\" (UniqueName: \"kubernetes.io/projected/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-kube-api-access-crqhh\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497738 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-run-k8s-cni-cncf-io\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.499298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497754 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a11e036-f7b7-4856-ad1a-96a0872133e5-hosts-file\") pod \"node-resolver-jfqm6\" (UID: \"1a11e036-f7b7-4856-ad1a-96a0872133e5\") " pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497770 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-kubelet-config\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.497917 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-run-systemd\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498034 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498053 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-daemon-config\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498094 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-cni-netd\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498113 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498137 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-host-cni-bin\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498149 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-host-run-k8s-cni-cncf-io\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498197 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-var-lib-openvswitch\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498224 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-run-ovn\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498246 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-conf-dir\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498294 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-run-ovn\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498315 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-registration-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498318 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31af1687-310d-4c90-9c64-63c8639315f4-var-lib-openvswitch\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498341 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-device-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498375 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-multus-conf-dir\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.500202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498379 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-hostroot\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.501055 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498417 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2232257f-864c-4500-b080-7bcdf2a062f1-host-slash\") pod \"iptables-alerter-qqjvc\" (UID: \"2232257f-864c-4500-b080-7bcdf2a062f1\") " pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.501055 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.498479 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8337463-d34f-4d10-9ed5-7819cf4c247a-hostroot\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.501941 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.501917 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31af1687-310d-4c90-9c64-63c8639315f4-ovn-node-metrics-cert\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.504229 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.504206 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:20:16.504327 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.504233 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:20:16.504327 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.504246 2584 projected.go:196] Error preparing data for projected volume kube-api-access-xwblm for pod openshift-network-diagnostics/network-check-target-gwpjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:16.504327 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.504313 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm podName:71bc58ba-c381-420f-aa47-c298bda76828 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:17.004294095 +0000 UTC m=+2.972106336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xwblm" (UniqueName: "kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm") pod "network-check-target-gwpjk" (UID: "71bc58ba-c381-420f-aa47-c298bda76828") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:16.506752 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.506697 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqpv\" (UniqueName: \"kubernetes.io/projected/31af1687-310d-4c90-9c64-63c8639315f4-kube-api-access-ptqpv\") pod \"ovnkube-node-g9q29\" (UID: \"31af1687-310d-4c90-9c64-63c8639315f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.506752 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.506746 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn965\" (UniqueName: \"kubernetes.io/projected/29e94806-d52b-476f-bf6b-11fab94483a6-kube-api-access-tn965\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:16.507900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.507874 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crqhh\" (UniqueName: \"kubernetes.io/projected/64558c33-88ec-4ebb-ae8d-56d19bcb7c07-kube-api-access-crqhh\") pod \"multus-additional-cni-plugins-m4tb5\" (UID: \"64558c33-88ec-4ebb-ae8d-56d19bcb7c07\") " pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.509311 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.509266 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdwv\" (UniqueName: \"kubernetes.io/projected/a8337463-d34f-4d10-9ed5-7819cf4c247a-kube-api-access-4zdwv\") pod \"multus-vmbsk\" (UID: \"a8337463-d34f-4d10-9ed5-7819cf4c247a\") " pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.598741 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-lib-modules\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.598798 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-sys-fs\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.598827 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-kubernetes\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.598850 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-run\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.598873 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-var-lib-kubelet\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.598896 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8866770d-bbc0-47e6-8007-06762e2c140f-etc-tuned\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.598922 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km7bq\" (UniqueName: \"kubernetes.io/projected/2232257f-864c-4500-b080-7bcdf2a062f1-kube-api-access-km7bq\") pod \"iptables-alerter-qqjvc\" (UID: \"2232257f-864c-4500-b080-7bcdf2a062f1\") " pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.598949 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkr2\" (UniqueName: \"kubernetes.io/projected/8866770d-bbc0-47e6-8007-06762e2c140f-kube-api-access-nzkr2\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.598975 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-modprobe-d\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.598997 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-host\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599021 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tksk\" (UniqueName: \"kubernetes.io/projected/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-kube-api-access-2tksk\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599052 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d45fd229-64fe-4f71-a53f-d20d4591c8df-konnectivity-ca\") pod \"konnectivity-agent-f89b2\" (UID: \"d45fd229-64fe-4f71-a53f-d20d4591c8df\") " pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599075 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-sysctl-d\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599116 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-sysctl-conf\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599139 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-systemd\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599168 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a11e036-f7b7-4856-ad1a-96a0872133e5-hosts-file\") pod \"node-resolver-jfqm6\" (UID: \"1a11e036-f7b7-4856-ad1a-96a0872133e5\") " pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:16.600506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599195 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-kubelet-config\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599225 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-registration-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599248 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-device-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599274 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2232257f-864c-4500-b080-7bcdf2a062f1-host-slash\") pod \"iptables-alerter-qqjvc\" (UID: \"2232257f-864c-4500-b080-7bcdf2a062f1\") " pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599311 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8866770d-bbc0-47e6-8007-06762e2c140f-tmp\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599358 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzjh\" (UniqueName: \"kubernetes.io/projected/2ed20544-9b54-4e94-9ebb-7cf426aabff9-kube-api-access-7vzjh\") pod \"node-ca-jzlhg\" (UID: \"2ed20544-9b54-4e94-9ebb-7cf426aabff9\") " pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599384 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-dbus\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599409 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599436 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqc7c\" (UniqueName: \"kubernetes.io/projected/1a11e036-f7b7-4856-ad1a-96a0872133e5-kube-api-access-sqc7c\") pod \"node-resolver-jfqm6\" (UID: \"1a11e036-f7b7-4856-ad1a-96a0872133e5\") " pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599460 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ed20544-9b54-4e94-9ebb-7cf426aabff9-serviceca\") pod \"node-ca-jzlhg\" (UID: \"2ed20544-9b54-4e94-9ebb-7cf426aabff9\") " pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599484 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d45fd229-64fe-4f71-a53f-d20d4591c8df-agent-certs\") pod \"konnectivity-agent-f89b2\" (UID: \"d45fd229-64fe-4f71-a53f-d20d4591c8df\") " pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599508 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-sys\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599533 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599568 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-socket-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599593 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ed20544-9b54-4e94-9ebb-7cf426aabff9-host\") pod \"node-ca-jzlhg\" (UID: \"2ed20544-9b54-4e94-9ebb-7cf426aabff9\") " pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599633 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a11e036-f7b7-4856-ad1a-96a0872133e5-tmp-dir\") pod \"node-resolver-jfqm6\" (UID: \"1a11e036-f7b7-4856-ad1a-96a0872133e5\") " pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599662 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2232257f-864c-4500-b080-7bcdf2a062f1-iptables-alerter-script\") pod \"iptables-alerter-qqjvc\" (UID: \"2232257f-864c-4500-b080-7bcdf2a062f1\") " pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.601611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599704 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-sysconfig\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599728 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599842 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.599973 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-lib-modules\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600030 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-sys-fs\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600041 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-registration-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600092 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-device-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600095 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-kubernetes\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600142 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-run\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600178 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2232257f-864c-4500-b080-7bcdf2a062f1-host-slash\") pod \"iptables-alerter-qqjvc\" (UID: \"2232257f-864c-4500-b080-7bcdf2a062f1\") " pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600219 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-var-lib-kubelet\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600377 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-systemd\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600527 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-sysctl-d\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600670 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-sysctl-conf\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600731 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a11e036-f7b7-4856-ad1a-96a0872133e5-hosts-file\") pod \"node-resolver-jfqm6\" (UID: \"1a11e036-f7b7-4856-ad1a-96a0872133e5\") " pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600789 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-kubelet-config\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600836 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d45fd229-64fe-4f71-a53f-d20d4591c8df-konnectivity-ca\") pod \"konnectivity-agent-f89b2\" (UID: \"d45fd229-64fe-4f71-a53f-d20d4591c8df\") " pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600865 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-modprobe-d\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.602481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.600912 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-host\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.601222 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-sys\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.601239 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ed20544-9b54-4e94-9ebb-7cf426aabff9-host\") pod \"node-ca-jzlhg\" (UID: \"2ed20544-9b54-4e94-9ebb-7cf426aabff9\") " pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.601279 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.601386 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-socket-dir\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.601479 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-dbus\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.601566 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:16.601622 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret podName:9dae04ef-d3ba-4c10-8fc6-9eef8093c500 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:17.101604011 +0000 UTC m=+3.069416244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret") pod "global-pull-secret-syncer-jclqb" (UID: "9dae04ef-d3ba-4c10-8fc6-9eef8093c500") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.601757 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2232257f-864c-4500-b080-7bcdf2a062f1-iptables-alerter-script\") pod \"iptables-alerter-qqjvc\" (UID: \"2232257f-864c-4500-b080-7bcdf2a062f1\") " pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.601977 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a11e036-f7b7-4856-ad1a-96a0872133e5-tmp-dir\") pod \"node-resolver-jfqm6\" (UID: \"1a11e036-f7b7-4856-ad1a-96a0872133e5\") " pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.602229 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ed20544-9b54-4e94-9ebb-7cf426aabff9-serviceca\") pod \"node-ca-jzlhg\" (UID: \"2ed20544-9b54-4e94-9ebb-7cf426aabff9\") " pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.603321 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.602242 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8866770d-bbc0-47e6-8007-06762e2c140f-etc-sysconfig\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.605854 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.605707 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8866770d-bbc0-47e6-8007-06762e2c140f-etc-tuned\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.606618 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.606588 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d45fd229-64fe-4f71-a53f-d20d4591c8df-agent-certs\") pod \"konnectivity-agent-f89b2\" (UID: \"d45fd229-64fe-4f71-a53f-d20d4591c8df\") " pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:16.607466 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.607419 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8866770d-bbc0-47e6-8007-06762e2c140f-tmp\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.612567 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.612542 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tksk\" (UniqueName: \"kubernetes.io/projected/2a0b6cf8-b740-4d9a-89d4-61ca036fc9af-kube-api-access-2tksk\") pod \"aws-ebs-csi-driver-node-8dbls\" (UID: \"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.612674 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.612592 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km7bq\" (UniqueName: \"kubernetes.io/projected/2232257f-864c-4500-b080-7bcdf2a062f1-kube-api-access-km7bq\") pod \"iptables-alerter-qqjvc\" (UID: \"2232257f-864c-4500-b080-7bcdf2a062f1\") " pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.618148 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.618121 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkr2\" (UniqueName: \"kubernetes.io/projected/8866770d-bbc0-47e6-8007-06762e2c140f-kube-api-access-nzkr2\") pod \"tuned-d9bxz\" (UID: \"8866770d-bbc0-47e6-8007-06762e2c140f\") " pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.618691 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.618638 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqc7c\" (UniqueName: \"kubernetes.io/projected/1a11e036-f7b7-4856-ad1a-96a0872133e5-kube-api-access-sqc7c\") pod \"node-resolver-jfqm6\" (UID: \"1a11e036-f7b7-4856-ad1a-96a0872133e5\") " pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:16.619296 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.619254 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzjh\" (UniqueName: \"kubernetes.io/projected/2ed20544-9b54-4e94-9ebb-7cf426aabff9-kube-api-access-7vzjh\") pod \"node-ca-jzlhg\" (UID: \"2ed20544-9b54-4e94-9ebb-7cf426aabff9\") " pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.692311 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.692007 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:16.708039 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.708007 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" Sep 10 19:20:16.718167 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.718146 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vmbsk" Sep 10 19:20:16.723434 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.723408 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:16.734803 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.734775 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" Sep 10 19:20:16.741717 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.741693 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" Sep 10 19:20:16.752685 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.752528 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jzlhg" Sep 10 19:20:16.762736 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.762714 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qqjvc" Sep 10 19:20:16.770675 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:16.770636 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jfqm6" Sep 10 19:20:17.003088 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:17.003007 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:17.003237 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:17.003114 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:17.003237 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:17.003166 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs podName:29e94806-d52b-476f-bf6b-11fab94483a6 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:18.003153045 +0000 UTC m=+3.970965271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs") pod "network-metrics-daemon-4fqpp" (UID: "29e94806-d52b-476f-bf6b-11fab94483a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:17.103752 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:17.103719 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:17.103933 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:17.103788 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwblm\" (UniqueName: \"kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm\") pod \"network-check-target-gwpjk\" (UID: \"71bc58ba-c381-420f-aa47-c298bda76828\") " pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:17.103933 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:17.103893 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:17.103933 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:17.103914 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:20:17.103933 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:17.103930 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:20:17.104120 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:17.103940 2584 projected.go:196] Error preparing data for projected volume kube-api-access-xwblm for pod openshift-network-diagnostics/network-check-target-gwpjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:17.104120 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:17.103974 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret podName:9dae04ef-d3ba-4c10-8fc6-9eef8093c500 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:18.103951672 +0000 UTC m=+4.071763904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret") pod "global-pull-secret-syncer-jclqb" (UID: "9dae04ef-d3ba-4c10-8fc6-9eef8093c500") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:17.104120 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:17.103995 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm podName:71bc58ba-c381-420f-aa47-c298bda76828 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:18.103984422 +0000 UTC m=+4.071796652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwblm" (UniqueName: "kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm") pod "network-check-target-gwpjk" (UID: "71bc58ba-c381-420f-aa47-c298bda76828") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:17.430498 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:17.430373 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:15:16 +0000 UTC" deadline="2028-06-03 11:13:47.862789568 +0000 UTC" Sep 10 19:20:17.430498 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:17.430422 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15159h53m30.432371267s" Sep 10 19:20:17.482723 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:17.482686 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:17.482896 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:17.482823 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:17.754984 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:17.754942 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8866770d_bbc0_47e6_8007_06762e2c140f.slice/crio-4ec02e7cc64f9918f4cd17c81ea7cad3f07512a23e5b61450c591eed4dffa8b4 WatchSource:0}: Error finding container 4ec02e7cc64f9918f4cd17c81ea7cad3f07512a23e5b61450c591eed4dffa8b4: Status 404 returned error can't find the container with id 4ec02e7cc64f9918f4cd17c81ea7cad3f07512a23e5b61450c591eed4dffa8b4 Sep 10 19:20:17.765137 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:17.764625 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a11e036_f7b7_4856_ad1a_96a0872133e5.slice/crio-ba803267b9ce07b057aa4a7c4b3c34f76e2a2900c44606f66f668200de27f53a WatchSource:0}: Error finding container ba803267b9ce07b057aa4a7c4b3c34f76e2a2900c44606f66f668200de27f53a: Status 404 returned error can't find the container with id ba803267b9ce07b057aa4a7c4b3c34f76e2a2900c44606f66f668200de27f53a Sep 10 19:20:17.767730 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:17.767703 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8337463_d34f_4d10_9ed5_7819cf4c247a.slice/crio-05f2344b94ea575f064c52e02b3511d42f513400e46772a8ac5f2b71e77698cf WatchSource:0}: Error finding container 05f2344b94ea575f064c52e02b3511d42f513400e46772a8ac5f2b71e77698cf: Status 404 returned error can't find the container with id 05f2344b94ea575f064c52e02b3511d42f513400e46772a8ac5f2b71e77698cf Sep 10 19:20:17.768615 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:17.768592 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45fd229_64fe_4f71_a53f_d20d4591c8df.slice/crio-8b96ebb798cbecf3e7c366e66653df02c360a09e320de4ffb7422daa99ef9b1a WatchSource:0}: Error finding container 8b96ebb798cbecf3e7c366e66653df02c360a09e320de4ffb7422daa99ef9b1a: Status 404 returned error can't find the container with id 8b96ebb798cbecf3e7c366e66653df02c360a09e320de4ffb7422daa99ef9b1a Sep 10 19:20:17.770419 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:17.770351 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31af1687_310d_4c90_9c64_63c8639315f4.slice/crio-ef2ee8c05d685631dd29d81fab82efcac5dd4a70068497aa266f2b2481bf031b WatchSource:0}: Error finding container ef2ee8c05d685631dd29d81fab82efcac5dd4a70068497aa266f2b2481bf031b: Status 404 returned error can't find the container with id ef2ee8c05d685631dd29d81fab82efcac5dd4a70068497aa266f2b2481bf031b Sep 10 19:20:17.771250 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:17.771205 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64558c33_88ec_4ebb_ae8d_56d19bcb7c07.slice/crio-d40301ea63ee54f6c50fb29d95de3a1eeeb58c7881b76530bbb6252ac377c44f WatchSource:0}: Error finding container d40301ea63ee54f6c50fb29d95de3a1eeeb58c7881b76530bbb6252ac377c44f: Status 404 returned error can't find the container with id d40301ea63ee54f6c50fb29d95de3a1eeeb58c7881b76530bbb6252ac377c44f Sep 10 19:20:17.772115 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:17.772087 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed20544_9b54_4e94_9ebb_7cf426aabff9.slice/crio-3dbb5f2e6bb1b56723c98c2ee1b695698625e311c2a313c44e44ee6820deacb3 WatchSource:0}: Error finding container 3dbb5f2e6bb1b56723c98c2ee1b695698625e311c2a313c44e44ee6820deacb3: Status 404 returned error can't find the container with id 3dbb5f2e6bb1b56723c98c2ee1b695698625e311c2a313c44e44ee6820deacb3 Sep 10 19:20:17.773850 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:17.773792 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0b6cf8_b740_4d9a_89d4_61ca036fc9af.slice/crio-5bf09cd1f0fe04c3d6b3f498a5711037298075b8bd548f971e260184000ea23d WatchSource:0}: Error finding container 5bf09cd1f0fe04c3d6b3f498a5711037298075b8bd548f971e260184000ea23d: Status 404 returned error can't find the container with id 5bf09cd1f0fe04c3d6b3f498a5711037298075b8bd548f971e260184000ea23d Sep 10 19:20:18.009655 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.009627 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:18.009807 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:18.009756 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:18.009866 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:18.009824 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs podName:29e94806-d52b-476f-bf6b-11fab94483a6 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:20.009804881 +0000 UTC m=+5.977617125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs") pod "network-metrics-daemon-4fqpp" (UID: "29e94806-d52b-476f-bf6b-11fab94483a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:18.111025 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.110993 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwblm\" (UniqueName: \"kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm\") pod \"network-check-target-gwpjk\" (UID: \"71bc58ba-c381-420f-aa47-c298bda76828\") " pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:18.111152 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.111046 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:18.111152 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:18.111140 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:20:18.111247 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:18.111160 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:20:18.111247 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:18.111165 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:18.111247 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:18.111170 2584 projected.go:196] Error preparing data for projected volume kube-api-access-xwblm for pod openshift-network-diagnostics/network-check-target-gwpjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:18.111247 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:18.111223 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret podName:9dae04ef-d3ba-4c10-8fc6-9eef8093c500 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:20.111203339 +0000 UTC m=+6.079015572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret") pod "global-pull-secret-syncer-jclqb" (UID: "9dae04ef-d3ba-4c10-8fc6-9eef8093c500") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:18.111247 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:18.111242 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm podName:71bc58ba-c381-420f-aa47-c298bda76828 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:20.111234813 +0000 UTC m=+6.079047039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwblm" (UniqueName: "kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm") pod "network-check-target-gwpjk" (UID: "71bc58ba-c381-420f-aa47-c298bda76828") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:18.431406 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.431271 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:15:16 +0000 UTC" deadline="2028-02-26 11:51:06.645486369 +0000 UTC" Sep 10 19:20:18.431406 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.431344 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12808h30m48.214146121s" Sep 10 19:20:18.487964 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.487214 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:18.487964 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:18.487402 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:18.487964 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.487814 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:18.487964 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:18.487906 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:18.503521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.502864 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal" event={"ID":"a45295e4379262bb71d3ff91072070b1","Type":"ContainerStarted","Data":"2decb711fb7d6b5df3b492c4c0860f69beff12c6e36a884ddbccdd340e83d4ae"} Sep 10 19:20:18.508500 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.508469 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qqjvc" event={"ID":"2232257f-864c-4500-b080-7bcdf2a062f1","Type":"ContainerStarted","Data":"30e8358fe964bd125805885081abbeaeda86bf708ff07d75aab31d04831c8335"} Sep 10 19:20:18.515796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.515721 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" event={"ID":"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af","Type":"ContainerStarted","Data":"5bf09cd1f0fe04c3d6b3f498a5711037298075b8bd548f971e260184000ea23d"} Sep 10 19:20:18.518328 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.518304 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jzlhg" event={"ID":"2ed20544-9b54-4e94-9ebb-7cf426aabff9","Type":"ContainerStarted","Data":"3dbb5f2e6bb1b56723c98c2ee1b695698625e311c2a313c44e44ee6820deacb3"} Sep 10 19:20:18.523633 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.523569 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" event={"ID":"64558c33-88ec-4ebb-ae8d-56d19bcb7c07","Type":"ContainerStarted","Data":"d40301ea63ee54f6c50fb29d95de3a1eeeb58c7881b76530bbb6252ac377c44f"} Sep 10 19:20:18.538361 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.533700 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" event={"ID":"31af1687-310d-4c90-9c64-63c8639315f4","Type":"ContainerStarted","Data":"ef2ee8c05d685631dd29d81fab82efcac5dd4a70068497aa266f2b2481bf031b"} Sep 10 19:20:18.538361 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.534839 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vmbsk" event={"ID":"a8337463-d34f-4d10-9ed5-7819cf4c247a","Type":"ContainerStarted","Data":"05f2344b94ea575f064c52e02b3511d42f513400e46772a8ac5f2b71e77698cf"} Sep 10 19:20:18.543459 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.543382 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jfqm6" event={"ID":"1a11e036-f7b7-4856-ad1a-96a0872133e5","Type":"ContainerStarted","Data":"ba803267b9ce07b057aa4a7c4b3c34f76e2a2900c44606f66f668200de27f53a"} Sep 10 19:20:18.554102 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.554073 2584 generic.go:356] "Generic (PLEG): container finished" podID="0af987b696cdc76ac0230a918c83b621" containerID="977c88c835ddb07a6afd3a197340f8c205a3a8fb25b01a027b390447e400a45f" exitCode=0 Sep 10 19:20:18.554211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.554157 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" event={"ID":"0af987b696cdc76ac0230a918c83b621","Type":"ContainerDied","Data":"977c88c835ddb07a6afd3a197340f8c205a3a8fb25b01a027b390447e400a45f"} Sep 10 19:20:18.565628 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.565564 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f89b2" event={"ID":"d45fd229-64fe-4f71-a53f-d20d4591c8df","Type":"ContainerStarted","Data":"8b96ebb798cbecf3e7c366e66653df02c360a09e320de4ffb7422daa99ef9b1a"} Sep 10 19:20:18.567988 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.567905 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" event={"ID":"8866770d-bbc0-47e6-8007-06762e2c140f","Type":"ContainerStarted","Data":"4ec02e7cc64f9918f4cd17c81ea7cad3f07512a23e5b61450c591eed4dffa8b4"} Sep 10 19:20:18.572239 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:18.572190 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-64.ec2.internal" podStartSLOduration=3.572174087 podStartE2EDuration="3.572174087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:18.521285665 +0000 UTC m=+4.489097917" watchObservedRunningTime="2026-09-10 19:20:18.572174087 +0000 UTC m=+4.539986338" Sep 10 19:20:19.482835 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:19.482805 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:19.483563 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:19.482943 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:19.606020 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:19.605985 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" event={"ID":"0af987b696cdc76ac0230a918c83b621","Type":"ContainerStarted","Data":"c69e1ac5fec44c4d74d067b7c12ef5789f63e03cc2c3b49fdbfa4c047b66dbfc"} Sep 10 19:20:20.032520 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:20.032486 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:20.032725 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:20.032632 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:20.032725 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:20.032693 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs podName:29e94806-d52b-476f-bf6b-11fab94483a6 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:24.032673625 +0000 UTC m=+10.000485857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs") pod "network-metrics-daemon-4fqpp" (UID: "29e94806-d52b-476f-bf6b-11fab94483a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:20.132792 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:20.132756 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:20.132959 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:20.132889 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:20.132959 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:20.132929 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwblm\" (UniqueName: \"kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm\") pod \"network-check-target-gwpjk\" (UID: \"71bc58ba-c381-420f-aa47-c298bda76828\") " pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:20.133088 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:20.132970 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret podName:9dae04ef-d3ba-4c10-8fc6-9eef8093c500 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:24.13294855 +0000 UTC m=+10.100760778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret") pod "global-pull-secret-syncer-jclqb" (UID: "9dae04ef-d3ba-4c10-8fc6-9eef8093c500") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:20.133154 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:20.133084 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:20:20.133154 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:20.133110 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:20:20.133154 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:20.133123 2584 projected.go:196] Error preparing data for projected volume kube-api-access-xwblm for pod openshift-network-diagnostics/network-check-target-gwpjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:20.133313 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:20.133173 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm podName:71bc58ba-c381-420f-aa47-c298bda76828 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:24.133155654 +0000 UTC m=+10.100967889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwblm" (UniqueName: "kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm") pod "network-check-target-gwpjk" (UID: "71bc58ba-c381-420f-aa47-c298bda76828") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:20.484804 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:20.484718 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:20.485265 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:20.484861 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:20.485265 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:20.485240 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:20.485407 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:20.485330 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:21.482356 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:21.482318 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:21.482546 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:21.482509 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:22.488254 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:22.488225 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:22.488683 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:22.488433 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:22.488683 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:22.488532 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:22.488683 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:22.488653 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:23.482336 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:23.482248 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:23.482488 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:23.482383 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:24.063473 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:24.063359 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:24.063909 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:24.063519 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:24.063909 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:24.063596 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs podName:29e94806-d52b-476f-bf6b-11fab94483a6 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:32.063574454 +0000 UTC m=+18.031386706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs") pod "network-metrics-daemon-4fqpp" (UID: "29e94806-d52b-476f-bf6b-11fab94483a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:24.164671 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:24.164640 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:24.164808 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:24.164703 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwblm\" (UniqueName: \"kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm\") pod \"network-check-target-gwpjk\" (UID: \"71bc58ba-c381-420f-aa47-c298bda76828\") " pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:24.164851 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:24.164803 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:24.164851 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:24.164821 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:20:24.164851 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:24.164835 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:20:24.164851 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:24.164845 2584 projected.go:196] Error preparing data for projected volume kube-api-access-xwblm for pod openshift-network-diagnostics/network-check-target-gwpjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:24.165036 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:24.164876 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret podName:9dae04ef-d3ba-4c10-8fc6-9eef8093c500 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:32.164853991 +0000 UTC m=+18.132666224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret") pod "global-pull-secret-syncer-jclqb" (UID: "9dae04ef-d3ba-4c10-8fc6-9eef8093c500") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:24.165036 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:24.164933 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm podName:71bc58ba-c381-420f-aa47-c298bda76828 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:32.164922083 +0000 UTC m=+18.132734313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwblm" (UniqueName: "kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm") pod "network-check-target-gwpjk" (UID: "71bc58ba-c381-420f-aa47-c298bda76828") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:24.483305 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:24.483207 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:24.483461 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:24.483332 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:24.483619 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:24.483590 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:24.483744 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:24.483707 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:25.482881 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:25.482770 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:25.482881 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:25.482885 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:26.482924 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:26.482894 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:26.483400 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:26.483024 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:26.483400 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:26.483388 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:26.483544 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:26.483486 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:27.482129 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:27.482091 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:27.482326 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:27.482224 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:28.482777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:28.482689 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:28.483243 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:28.482834 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:28.483243 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:28.482878 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:28.483243 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:28.482992 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:29.482709 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:29.482497 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:29.482908 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:29.482815 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:30.482123 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:30.482085 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:30.482317 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:30.482083 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:30.482317 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:30.482211 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:30.482317 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:30.482260 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:31.482644 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:31.482610 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:31.483073 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:31.482724 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:32.126439 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:32.126395 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:32.126620 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:32.126520 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:32.126620 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:32.126580 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs podName:29e94806-d52b-476f-bf6b-11fab94483a6 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:48.126566473 +0000 UTC m=+34.094378698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs") pod "network-metrics-daemon-4fqpp" (UID: "29e94806-d52b-476f-bf6b-11fab94483a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:32.227600 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:32.227543 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:32.227766 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:32.227646 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwblm\" (UniqueName: \"kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm\") pod \"network-check-target-gwpjk\" (UID: \"71bc58ba-c381-420f-aa47-c298bda76828\") " pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:32.227766 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:32.227704 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:32.227877 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:32.227779 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:20:32.227877 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:32.227787 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret podName:9dae04ef-d3ba-4c10-8fc6-9eef8093c500 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:48.227766142 +0000 UTC m=+34.195578387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret") pod "global-pull-secret-syncer-jclqb" (UID: "9dae04ef-d3ba-4c10-8fc6-9eef8093c500") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:32.227877 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:32.227799 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:20:32.227877 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:32.227813 2584 projected.go:196] Error preparing data for projected volume kube-api-access-xwblm for pod openshift-network-diagnostics/network-check-target-gwpjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:32.227877 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:32.227866 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm podName:71bc58ba-c381-420f-aa47-c298bda76828 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:48.22785031 +0000 UTC m=+34.195662538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwblm" (UniqueName: "kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm") pod "network-check-target-gwpjk" (UID: "71bc58ba-c381-420f-aa47-c298bda76828") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:32.482125 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:32.482039 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:32.482314 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:32.482038 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:32.482314 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:32.482175 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:32.482314 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:32.482245 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:33.482745 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:33.482704 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:33.483191 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:33.482858 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:34.482876 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:34.482819 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:34.483315 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:34.482966 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:34.483315 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:34.483006 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:34.483315 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:34.483102 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:35.481943 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:35.481904 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:35.482099 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:35.482013 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:36.482326 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.482114 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:36.483081 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.482117 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:36.483081 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:36.482443 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:36.483081 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:36.482500 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:36.655818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.655778 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" event={"ID":"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af","Type":"ContainerStarted","Data":"2b5dacb3409dcf0d0804a5780a624261c8fb124e53b55153bb145732eb5e3ee9"} Sep 10 19:20:36.657643 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.657603 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jzlhg" event={"ID":"2ed20544-9b54-4e94-9ebb-7cf426aabff9","Type":"ContainerStarted","Data":"6ce4bdeea65f8909daeaea6d62e627ce513753d01261cbc70e0c1e9f21e80438"} Sep 10 19:20:36.659556 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.659452 2584 generic.go:356] "Generic (PLEG): container finished" podID="64558c33-88ec-4ebb-ae8d-56d19bcb7c07" containerID="4fb68fa4be032519e95e18f3da3158447495f61c40af513a38299b7c6136d22f" exitCode=0 Sep 10 19:20:36.659556 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.659487 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" event={"ID":"64558c33-88ec-4ebb-ae8d-56d19bcb7c07","Type":"ContainerDied","Data":"4fb68fa4be032519e95e18f3da3158447495f61c40af513a38299b7c6136d22f"} Sep 10 19:20:36.663338 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.663311 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" event={"ID":"31af1687-310d-4c90-9c64-63c8639315f4","Type":"ContainerStarted","Data":"7f617d93b801708a33fcb11778f6e6a3e7a57c55ef18f642b25b7d5a5cbe120b"} Sep 10 19:20:36.663453 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.663411 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" event={"ID":"31af1687-310d-4c90-9c64-63c8639315f4","Type":"ContainerStarted","Data":"419557c76e4d2d76ce5fc371c978163641a6debfa683aaa0aba35d93a8dcc518"} Sep 10 19:20:36.663503 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.663460 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" event={"ID":"31af1687-310d-4c90-9c64-63c8639315f4","Type":"ContainerStarted","Data":"8c306bd492286358d65207efb112662079f262f26a17697f36600bc4ddf1fc90"} Sep 10 19:20:36.663545 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.663511 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" event={"ID":"31af1687-310d-4c90-9c64-63c8639315f4","Type":"ContainerStarted","Data":"067cecf2ea67b98d167c17a1c045cd62eac2da9a4ce9807274375cce83482174"} Sep 10 19:20:36.663599 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.663551 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" event={"ID":"31af1687-310d-4c90-9c64-63c8639315f4","Type":"ContainerStarted","Data":"022edb95aa1137aeb67f5d5f02dbf989039b4e2ff8c3dcd5c35d128c376c6e5d"} Sep 10 19:20:36.663636 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.663597 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" event={"ID":"31af1687-310d-4c90-9c64-63c8639315f4","Type":"ContainerStarted","Data":"df8187c2eaef3655625225f5dc50c947cc7539759da5a6e17320e0239c60fc5c"} Sep 10 19:20:36.665099 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.665078 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vmbsk" event={"ID":"a8337463-d34f-4d10-9ed5-7819cf4c247a","Type":"ContainerStarted","Data":"ec0707c8f75b6a8356440815369d1bced4f1ba5ec8cc921af033737fa39714a3"} Sep 10 19:20:36.666838 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.666818 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jfqm6" event={"ID":"1a11e036-f7b7-4856-ad1a-96a0872133e5","Type":"ContainerStarted","Data":"e113f8d2d3804dc7d8833052833b72b2a0687618f748636f984fa751fa8b04fc"} Sep 10 19:20:36.668662 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.668636 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f89b2" event={"ID":"d45fd229-64fe-4f71-a53f-d20d4591c8df","Type":"ContainerStarted","Data":"c6dd2c2c9372f3ff97de34ba00af2d36de12691b445f8c847774f7b7e3b98668"} Sep 10 19:20:36.671992 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.671949 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jzlhg" podStartSLOduration=3.82649477 podStartE2EDuration="21.671933938s" totalImagesPullingTime="17.845439168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:36.671738845 +0000 UTC m=+22.639551107" watchObservedRunningTime="2026-09-10 19:20:36.671933938 +0000 UTC m=+22.639746185" Sep 10 19:20:36.672344 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.672308 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-64.ec2.internal" podStartSLOduration=21.67229974 podStartE2EDuration="21.67229974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:19.622396653 +0000 UTC m=+5.590208906" watchObservedRunningTime="2026-09-10 19:20:36.67229974 +0000 UTC m=+22.640111979" Sep 10 19:20:36.672604 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.672568 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" event={"ID":"8866770d-bbc0-47e6-8007-06762e2c140f","Type":"ContainerStarted","Data":"b9526cf66dbd4c0b1118bb332fd095e0a163f92bfa5d580b9c778370cd142058"} Sep 10 19:20:36.689783 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.689718 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vmbsk" podStartSLOduration=3.610957269 podStartE2EDuration="21.68969955s" totalImagesPullingTime="18.078742281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:36.689545825 +0000 UTC m=+22.657358063" watchObservedRunningTime="2026-09-10 19:20:36.68969955 +0000 UTC m=+22.657511796" Sep 10 19:20:36.708699 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.708643 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f89b2" podStartSLOduration=11.337743918 podStartE2EDuration="21.708625398s" totalImagesPullingTime="10.37088148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:36.708083882 +0000 UTC m=+22.675896132" watchObservedRunningTime="2026-09-10 19:20:36.708625398 +0000 UTC m=+22.676437645" Sep 10 19:20:36.723166 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.723103 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jfqm6" podStartSLOduration=3.88834179 podStartE2EDuration="21.723089289s" totalImagesPullingTime="17.834747499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:36.722793106 +0000 UTC m=+22.690605356" watchObservedRunningTime="2026-09-10 19:20:36.723089289 +0000 UTC m=+22.690901535" Sep 10 19:20:36.765866 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:36.765814 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d9bxz" podStartSLOduration=3.904700548 podStartE2EDuration="21.765800872s" totalImagesPullingTime="17.861100324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:36.765424619 +0000 UTC m=+22.733236866" watchObservedRunningTime="2026-09-10 19:20:36.765800872 +0000 UTC m=+22.733613118" Sep 10 19:20:37.221795 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:37.221759 2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 19:20:37.441556 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:37.441443 2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T19:20:37.221779147Z","UUID":"b4107840-fd8f-4844-bccd-3dd3c4784d36","Handler":null,"Name":"","Endpoint":""} Sep 10 19:20:37.443176 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:37.443154 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 19:20:37.443329 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:37.443189 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 19:20:37.482331 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:37.482238 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:37.482502 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:37.482392 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:37.677140 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:37.677103 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qqjvc" event={"ID":"2232257f-864c-4500-b080-7bcdf2a062f1","Type":"ContainerStarted","Data":"160147350ad1a998fc439a765152ee6c439b72b517454ec0eead223099472666"} Sep 10 19:20:37.679472 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:37.679438 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" event={"ID":"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af","Type":"ContainerStarted","Data":"5d75b330e2e6e9a67ba3271635acf5f63019877ce390a6d7859932ef4d689674"} Sep 10 19:20:37.692154 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:37.692091 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qqjvc" podStartSLOduration=4.852496889 podStartE2EDuration="22.692067827s" totalImagesPullingTime="17.839570938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:37.691623842 +0000 UTC m=+23.659436089" watchObservedRunningTime="2026-09-10 19:20:37.692067827 +0000 UTC m=+23.659880075" Sep 10 19:20:38.483364 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:38.482973 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:38.483364 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:38.483111 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:38.483364 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:38.483170 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:38.483364 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:38.483244 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:38.684547 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:38.684179 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" event={"ID":"2a0b6cf8-b740-4d9a-89d4-61ca036fc9af","Type":"ContainerStarted","Data":"4c7b90773db975aa78970d4ef49d8ce77a380b295e8ed591dc1b4aa85c50cb86"} Sep 10 19:20:38.688059 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:38.688010 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" event={"ID":"31af1687-310d-4c90-9c64-63c8639315f4","Type":"ContainerStarted","Data":"0bc060acf296539420239d53b54b771f3d62fa797d551076cfdc9144f81543a9"} Sep 10 19:20:38.703892 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:38.703838 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dbls" podStartSLOduration=3.427867877 podStartE2EDuration="23.703816314s" totalImagesPullingTime="20.275948437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:38.703677697 +0000 UTC m=+24.671489947" watchObservedRunningTime="2026-09-10 19:20:38.703816314 +0000 UTC m=+24.671628561" Sep 10 19:20:39.482437 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:39.482401 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:39.482642 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:39.482529 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:40.485660 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:40.485627 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:40.486191 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:40.485678 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:40.486191 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:40.485745 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:40.486191 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:40.485878 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:41.376687 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:41.376496 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:41.377083 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:41.377064 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:41.482094 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:41.482000 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:41.482240 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:41.482115 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:41.697349 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:41.697316 2584 generic.go:356] "Generic (PLEG): container finished" podID="64558c33-88ec-4ebb-ae8d-56d19bcb7c07" containerID="772c6022894c4dcfdf8de119a9f632ebf0dda7129b85e7a5b7592fd82364457b" exitCode=0 Sep 10 19:20:41.697847 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:41.697361 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" event={"ID":"64558c33-88ec-4ebb-ae8d-56d19bcb7c07","Type":"ContainerDied","Data":"772c6022894c4dcfdf8de119a9f632ebf0dda7129b85e7a5b7592fd82364457b"} Sep 10 19:20:41.702939 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:41.702911 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" event={"ID":"31af1687-310d-4c90-9c64-63c8639315f4","Type":"ContainerStarted","Data":"de39617502a2d174dab0e023a9d605a0127337be8fffa157c54f52206937a64d"} Sep 10 19:20:41.703110 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:41.703086 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:41.703525 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:41.703508 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f89b2" Sep 10 19:20:41.757985 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:41.757883 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" podStartSLOduration=9.892276266 podStartE2EDuration="27.757867426s" totalImagesPullingTime="17.86559116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:41.756572751 +0000 UTC m=+27.724384999" watchObservedRunningTime="2026-09-10 19:20:41.757867426 +0000 UTC m=+27.725679673" Sep 10 19:20:42.486888 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:42.486856 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:42.487147 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:42.487002 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:42.487147 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:42.487041 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:42.487147 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:42.487114 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:42.705324 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:42.705273 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:42.705879 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:42.705378 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:42.705879 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:42.705430 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:42.720758 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:42.720727 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:42.720890 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:42.720879 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:20:43.237963 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:43.237925 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gwpjk"] Sep 10 19:20:43.238122 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:43.238051 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:43.238178 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:43.238145 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:43.241481 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:43.241447 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jclqb"] Sep 10 19:20:43.241645 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:43.241577 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:43.241713 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:43.241682 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:43.242151 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:43.242127 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4fqpp"] Sep 10 19:20:43.242273 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:43.242216 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:43.242362 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:43.242326 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:43.709732 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:43.709542 2584 generic.go:356] "Generic (PLEG): container finished" podID="64558c33-88ec-4ebb-ae8d-56d19bcb7c07" containerID="1d09296ab631fa3efc888dcc425ae783af6e8c9ee49e002509df5c94c8004073" exitCode=0 Sep 10 19:20:43.710347 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:43.709624 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" event={"ID":"64558c33-88ec-4ebb-ae8d-56d19bcb7c07","Type":"ContainerDied","Data":"1d09296ab631fa3efc888dcc425ae783af6e8c9ee49e002509df5c94c8004073"} Sep 10 19:20:44.482817 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:44.482781 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:44.482817 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:44.482804 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:44.483053 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:44.482883 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:44.483053 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:44.482997 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:45.482242 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:45.482202 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:45.482703 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:45.482332 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:45.717755 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:45.717657 2584 generic.go:356] "Generic (PLEG): container finished" podID="64558c33-88ec-4ebb-ae8d-56d19bcb7c07" containerID="59c5538c9301fbb46e82f1c4ca7bd642f38df0f4f366b02974a938bda4c9f177" exitCode=0 Sep 10 19:20:45.717755 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:45.717722 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" event={"ID":"64558c33-88ec-4ebb-ae8d-56d19bcb7c07","Type":"ContainerDied","Data":"59c5538c9301fbb46e82f1c4ca7bd642f38df0f4f366b02974a938bda4c9f177"} Sep 10 19:20:46.482843 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:46.482804 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:46.483295 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:46.482804 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:46.483295 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:46.482936 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jclqb" podUID="9dae04ef-d3ba-4c10-8fc6-9eef8093c500" Sep 10 19:20:46.483295 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:46.482996 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4fqpp" podUID="29e94806-d52b-476f-bf6b-11fab94483a6" Sep 10 19:20:47.482889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:47.482853 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:47.483356 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:47.482991 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwpjk" podUID="71bc58ba-c381-420f-aa47-c298bda76828" Sep 10 19:20:48.145719 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.145683 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:48.145917 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:48.145802 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:48.145917 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:48.145856 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs podName:29e94806-d52b-476f-bf6b-11fab94483a6 nodeName:}" failed. No retries permitted until 2026-09-10 19:21:20.145841809 +0000 UTC m=+66.113654034 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs") pod "network-metrics-daemon-4fqpp" (UID: "29e94806-d52b-476f-bf6b-11fab94483a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:20:48.246365 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.246325 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwblm\" (UniqueName: \"kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm\") pod \"network-check-target-gwpjk\" (UID: \"71bc58ba-c381-420f-aa47-c298bda76828\") " pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:48.246520 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.246394 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:48.246520 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:48.246446 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:20:48.246520 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:48.246470 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:20:48.246520 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:48.246481 2584 projected.go:196] Error preparing data for projected volume kube-api-access-xwblm for pod openshift-network-diagnostics/network-check-target-gwpjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:48.246520 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:48.246504 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:48.246698 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:48.246544 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm podName:71bc58ba-c381-420f-aa47-c298bda76828 nodeName:}" failed. No retries permitted until 2026-09-10 19:21:20.246524902 +0000 UTC m=+66.214337126 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwblm" (UniqueName: "kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm") pod "network-check-target-gwpjk" (UID: "71bc58ba-c381-420f-aa47-c298bda76828") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:20:48.246698 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:48.246559 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret podName:9dae04ef-d3ba-4c10-8fc6-9eef8093c500 nodeName:}" failed. No retries permitted until 2026-09-10 19:21:20.24655298 +0000 UTC m=+66.214365206 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret") pod "global-pull-secret-syncer-jclqb" (UID: "9dae04ef-d3ba-4c10-8fc6-9eef8093c500") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:20:48.345570 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.345535 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-64.ec2.internal" event="NodeReady" Sep 10 19:20:48.345764 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.345690 2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 19:20:48.391099 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.391060 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-klkw2"] Sep 10 19:20:48.393841 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.393816 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.396824 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.396759 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 19:20:48.396952 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.396835 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wv5m9"] Sep 10 19:20:48.396952 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.396863 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2kts8\"" Sep 10 19:20:48.397353 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.397331 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 19:20:48.399083 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.399065 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wv5m9" Sep 10 19:20:48.401366 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.401346 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 19:20:48.401479 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.401428 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gl8x4\"" Sep 10 19:20:48.401883 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.401618 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 19:20:48.401883 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.401680 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 19:20:48.406247 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.406226 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-klkw2"] Sep 10 19:20:48.410866 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.410715 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wv5m9"] Sep 10 19:20:48.482143 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.482112 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:20:48.482329 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.482112 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:20:48.485072 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.485048 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npjmj\"" Sep 10 19:20:48.485435 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.485088 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 19:20:48.485435 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.485053 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 19:20:48.548458 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.548421 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.548458 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.548464 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7kk8\" (UniqueName: \"kubernetes.io/projected/974975b6-d004-4708-a3d3-1757891d57a0-kube-api-access-z7kk8\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.548717 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.548532 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f79489-278b-41b0-b002-474e6da0dc7c-cert\") pod \"ingress-canary-wv5m9\" (UID: \"23f79489-278b-41b0-b002-474e6da0dc7c\") " pod="openshift-ingress-canary/ingress-canary-wv5m9" Sep 10 19:20:48.548717 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.548603 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxjb\" (UniqueName: \"kubernetes.io/projected/23f79489-278b-41b0-b002-474e6da0dc7c-kube-api-access-fbxjb\") pod \"ingress-canary-wv5m9\" (UID: \"23f79489-278b-41b0-b002-474e6da0dc7c\") " pod="openshift-ingress-canary/ingress-canary-wv5m9" Sep 10 19:20:48.548717 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.548624 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/974975b6-d004-4708-a3d3-1757891d57a0-config-volume\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.548717 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.548702 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/974975b6-d004-4708-a3d3-1757891d57a0-tmp-dir\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.650070 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.649974 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxjb\" (UniqueName: \"kubernetes.io/projected/23f79489-278b-41b0-b002-474e6da0dc7c-kube-api-access-fbxjb\") pod \"ingress-canary-wv5m9\" (UID: \"23f79489-278b-41b0-b002-474e6da0dc7c\") " pod="openshift-ingress-canary/ingress-canary-wv5m9" Sep 10 19:20:48.650070 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.650025 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/974975b6-d004-4708-a3d3-1757891d57a0-config-volume\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.650314 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.650082 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/974975b6-d004-4708-a3d3-1757891d57a0-tmp-dir\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.650314 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.650122 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.650314 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.650151 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7kk8\" (UniqueName: \"kubernetes.io/projected/974975b6-d004-4708-a3d3-1757891d57a0-kube-api-access-z7kk8\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.650314 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.650201 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f79489-278b-41b0-b002-474e6da0dc7c-cert\") pod \"ingress-canary-wv5m9\" (UID: \"23f79489-278b-41b0-b002-474e6da0dc7c\") " pod="openshift-ingress-canary/ingress-canary-wv5m9" Sep 10 19:20:48.650314 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:48.650262 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:20:48.650554 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:48.650355 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls podName:974975b6-d004-4708-a3d3-1757891d57a0 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:49.150330427 +0000 UTC m=+35.118142665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls") pod "dns-default-klkw2" (UID: "974975b6-d004-4708-a3d3-1757891d57a0") : secret "dns-default-metrics-tls" not found Sep 10 19:20:48.650917 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.650700 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/974975b6-d004-4708-a3d3-1757891d57a0-config-volume\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.651168 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.651152 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/974975b6-d004-4708-a3d3-1757891d57a0-tmp-dir\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.654817 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.654794 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f79489-278b-41b0-b002-474e6da0dc7c-cert\") pod \"ingress-canary-wv5m9\" (UID: \"23f79489-278b-41b0-b002-474e6da0dc7c\") " pod="openshift-ingress-canary/ingress-canary-wv5m9" Sep 10 19:20:48.661987 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.661962 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxjb\" (UniqueName: \"kubernetes.io/projected/23f79489-278b-41b0-b002-474e6da0dc7c-kube-api-access-fbxjb\") pod \"ingress-canary-wv5m9\" (UID: \"23f79489-278b-41b0-b002-474e6da0dc7c\") " pod="openshift-ingress-canary/ingress-canary-wv5m9" Sep 10 19:20:48.662148 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.662130 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7kk8\" (UniqueName: \"kubernetes.io/projected/974975b6-d004-4708-a3d3-1757891d57a0-kube-api-access-z7kk8\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:48.714271 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:48.714220 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wv5m9" Sep 10 19:20:49.154300 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:49.154242 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:49.154505 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:49.154395 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:20:49.154505 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:49.154477 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls podName:974975b6-d004-4708-a3d3-1757891d57a0 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:50.154450864 +0000 UTC m=+36.122263099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls") pod "dns-default-klkw2" (UID: "974975b6-d004-4708-a3d3-1757891d57a0") : secret "dns-default-metrics-tls" not found Sep 10 19:20:49.481975 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:49.481893 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:20:49.485122 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:49.485086 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-plphg\"" Sep 10 19:20:49.485530 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:49.485385 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 19:20:49.485950 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:49.485930 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 19:20:49.898811 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:49.898776 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wv5m9"] Sep 10 19:20:49.902716 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:49.902681 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f79489_278b_41b0_b002_474e6da0dc7c.slice/crio-be847cf8f6de3d0eb6f21d678459da196144023de457ae2ad95f58c49dbf52f3 WatchSource:0}: Error finding container be847cf8f6de3d0eb6f21d678459da196144023de457ae2ad95f58c49dbf52f3: Status 404 returned error can't find the container with id be847cf8f6de3d0eb6f21d678459da196144023de457ae2ad95f58c49dbf52f3 Sep 10 19:20:50.161117 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:50.161078 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:50.161340 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:50.161247 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:20:50.161455 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:50.161351 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls podName:974975b6-d004-4708-a3d3-1757891d57a0 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:52.161334056 +0000 UTC m=+38.129146281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls") pod "dns-default-klkw2" (UID: "974975b6-d004-4708-a3d3-1757891d57a0") : secret "dns-default-metrics-tls" not found Sep 10 19:20:50.733048 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:50.733013 2584 generic.go:356] "Generic (PLEG): container finished" podID="64558c33-88ec-4ebb-ae8d-56d19bcb7c07" containerID="69b4172ec3dfcad42b6fb949c23488778c663b9dd635d6f1ad12a29bdd7988e0" exitCode=0 Sep 10 19:20:50.733980 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:50.733108 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" event={"ID":"64558c33-88ec-4ebb-ae8d-56d19bcb7c07","Type":"ContainerDied","Data":"69b4172ec3dfcad42b6fb949c23488778c663b9dd635d6f1ad12a29bdd7988e0"} Sep 10 19:20:50.734546 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:50.734522 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wv5m9" event={"ID":"23f79489-278b-41b0-b002-474e6da0dc7c","Type":"ContainerStarted","Data":"be847cf8f6de3d0eb6f21d678459da196144023de457ae2ad95f58c49dbf52f3"} Sep 10 19:20:51.740906 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:51.740685 2584 generic.go:356] "Generic (PLEG): container finished" podID="64558c33-88ec-4ebb-ae8d-56d19bcb7c07" containerID="6c56ca74799130b1ed80b3eb4f3115c1384d412b13ed0921a0772ac57a9d242f" exitCode=0 Sep 10 19:20:51.740906 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:51.740777 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" event={"ID":"64558c33-88ec-4ebb-ae8d-56d19bcb7c07","Type":"ContainerDied","Data":"6c56ca74799130b1ed80b3eb4f3115c1384d412b13ed0921a0772ac57a9d242f"} Sep 10 19:20:52.178621 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:52.178588 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:52.178757 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:52.178703 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:20:52.178757 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:52.178756 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls podName:974975b6-d004-4708-a3d3-1757891d57a0 nodeName:}" failed. No retries permitted until 2026-09-10 19:20:56.17874167 +0000 UTC m=+42.146553895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls") pod "dns-default-klkw2" (UID: "974975b6-d004-4708-a3d3-1757891d57a0") : secret "dns-default-metrics-tls" not found Sep 10 19:20:52.747055 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:52.747024 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" event={"ID":"64558c33-88ec-4ebb-ae8d-56d19bcb7c07","Type":"ContainerStarted","Data":"1217af436c691b5ccf0c70a5f162c1a6d4b6231caa1089cc13716b61f0bde60e"} Sep 10 19:20:52.749163 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:52.749128 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wv5m9" event={"ID":"23f79489-278b-41b0-b002-474e6da0dc7c","Type":"ContainerStarted","Data":"726dd10b616cf2fb9e05ce177ef64b185b8dbd19324c97315d9644f64b8d51e6"} Sep 10 19:20:52.771507 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:52.771456 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m4tb5" podStartSLOduration=9.330899247 podStartE2EDuration="37.771441235s" totalImagesPullingTime="28.440541988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:52.770143959 +0000 UTC m=+38.737956207" watchObservedRunningTime="2026-09-10 19:20:52.771441235 +0000 UTC m=+38.739253516" Sep 10 19:20:52.789355 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:52.789297 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wv5m9" podStartSLOduration=2.424877833 podStartE2EDuration="4.789260051s" totalImagesPullingTime="2.364382218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:52.788544969 +0000 UTC m=+38.756357216" watchObservedRunningTime="2026-09-10 19:20:52.789260051 +0000 UTC m=+38.757072298" Sep 10 19:20:53.583516 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.583476 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hzjrz"] Sep 10 19:20:53.607222 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.607193 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzjrz"] Sep 10 19:20:53.607377 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.607334 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.609631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.609610 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 19:20:53.610010 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.609989 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 19:20:53.610103 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.609990 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 19:20:53.610160 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.610135 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 19:20:53.610160 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.610141 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hkk9\"" Sep 10 19:20:53.689539 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.689498 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56353bf4-bd0a-4710-9f0e-35d21ddda90b-crio-socket\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.689714 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.689558 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.689714 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.689633 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.689714 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.689677 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56353bf4-bd0a-4710-9f0e-35d21ddda90b-data-volume\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.689813 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.689737 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcfzv\" (UniqueName: \"kubernetes.io/projected/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-api-access-jcfzv\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.790207 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.790163 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.790207 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.790216 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56353bf4-bd0a-4710-9f0e-35d21ddda90b-data-volume\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.790679 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:53.790301 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:20:53.790679 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.790349 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcfzv\" (UniqueName: \"kubernetes.io/projected/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-api-access-jcfzv\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.790679 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:53.790380 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm podName:56353bf4-bd0a-4710-9f0e-35d21ddda90b nodeName:}" failed. No retries permitted until 2026-09-10 19:20:54.29036395 +0000 UTC m=+40.258176176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzjrz" (UID: "56353bf4-bd0a-4710-9f0e-35d21ddda90b") : configmap "kube-rbac-proxy" not found Sep 10 19:20:53.790679 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.790430 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56353bf4-bd0a-4710-9f0e-35d21ddda90b-crio-socket\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.790679 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.790476 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.790679 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.790550 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56353bf4-bd0a-4710-9f0e-35d21ddda90b-data-volume\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.790679 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:53.790557 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:20:53.790679 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:53.790642 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls podName:56353bf4-bd0a-4710-9f0e-35d21ddda90b nodeName:}" failed. No retries permitted until 2026-09-10 19:20:54.290629246 +0000 UTC m=+40.258441484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjrz" (UID: "56353bf4-bd0a-4710-9f0e-35d21ddda90b") : secret "insights-runtime-extractor-tls" not found Sep 10 19:20:53.790958 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.790742 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56353bf4-bd0a-4710-9f0e-35d21ddda90b-crio-socket\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:53.799533 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:53.799489 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcfzv\" (UniqueName: \"kubernetes.io/projected/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-api-access-jcfzv\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:54.107578 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.107543 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v"] Sep 10 19:20:54.134120 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.134083 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v" Sep 10 19:20:54.135899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.135864 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v"] Sep 10 19:20:54.136919 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.136897 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ts2t9\"" Sep 10 19:20:54.137396 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.137375 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 19:20:54.137802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.137782 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 19:20:54.293908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.293865 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:54.293908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.293912 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:54.294213 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.293951 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brtf9\" (UniqueName: \"kubernetes.io/projected/9eab30f3-754a-47c2-a5e4-1c1e0faab235-kube-api-access-brtf9\") pod \"migrator-fdbb8fdff-d845v\" (UID: \"9eab30f3-754a-47c2-a5e4-1c1e0faab235\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v" Sep 10 19:20:54.294213 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:54.294003 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:20:54.294213 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:54.294056 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:20:54.294213 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:54.294076 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls podName:56353bf4-bd0a-4710-9f0e-35d21ddda90b nodeName:}" failed. No retries permitted until 2026-09-10 19:20:55.294059192 +0000 UTC m=+41.261871420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjrz" (UID: "56353bf4-bd0a-4710-9f0e-35d21ddda90b") : secret "insights-runtime-extractor-tls" not found Sep 10 19:20:54.294213 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:54.294105 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm podName:56353bf4-bd0a-4710-9f0e-35d21ddda90b nodeName:}" failed. No retries permitted until 2026-09-10 19:20:55.294092805 +0000 UTC m=+41.261905030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzjrz" (UID: "56353bf4-bd0a-4710-9f0e-35d21ddda90b") : configmap "kube-rbac-proxy" not found Sep 10 19:20:54.394530 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.394438 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brtf9\" (UniqueName: \"kubernetes.io/projected/9eab30f3-754a-47c2-a5e4-1c1e0faab235-kube-api-access-brtf9\") pod \"migrator-fdbb8fdff-d845v\" (UID: \"9eab30f3-754a-47c2-a5e4-1c1e0faab235\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v" Sep 10 19:20:54.402413 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.402386 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brtf9\" (UniqueName: \"kubernetes.io/projected/9eab30f3-754a-47c2-a5e4-1c1e0faab235-kube-api-access-brtf9\") pod \"migrator-fdbb8fdff-d845v\" (UID: \"9eab30f3-754a-47c2-a5e4-1c1e0faab235\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v" Sep 10 19:20:54.443405 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.443377 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v" Sep 10 19:20:54.731736 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.731659 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v"] Sep 10 19:20:54.735119 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:54.735089 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eab30f3_754a_47c2_a5e4_1c1e0faab235.slice/crio-f1868aae46c3a9df6088f916840d8caa6b5e69c8060a79818ca18cd384512694 WatchSource:0}: Error finding container f1868aae46c3a9df6088f916840d8caa6b5e69c8060a79818ca18cd384512694: Status 404 returned error can't find the container with id f1868aae46c3a9df6088f916840d8caa6b5e69c8060a79818ca18cd384512694 Sep 10 19:20:54.755356 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:54.755313 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v" event={"ID":"9eab30f3-754a-47c2-a5e4-1c1e0faab235","Type":"ContainerStarted","Data":"f1868aae46c3a9df6088f916840d8caa6b5e69c8060a79818ca18cd384512694"} Sep 10 19:20:55.300846 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:55.300802 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:55.301240 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:55.300853 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:55.301240 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:55.300959 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:20:55.301240 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:55.300981 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:20:55.301240 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:55.301034 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm podName:56353bf4-bd0a-4710-9f0e-35d21ddda90b nodeName:}" failed. No retries permitted until 2026-09-10 19:20:57.301019806 +0000 UTC m=+43.268832031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzjrz" (UID: "56353bf4-bd0a-4710-9f0e-35d21ddda90b") : configmap "kube-rbac-proxy" not found Sep 10 19:20:55.301240 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:55.301051 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls podName:56353bf4-bd0a-4710-9f0e-35d21ddda90b nodeName:}" failed. No retries permitted until 2026-09-10 19:20:57.301045549 +0000 UTC m=+43.268857773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjrz" (UID: "56353bf4-bd0a-4710-9f0e-35d21ddda90b") : secret "insights-runtime-extractor-tls" not found Sep 10 19:20:55.749777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:55.749737 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jfqm6_1a11e036-f7b7-4856-ad1a-96a0872133e5/dns-node-resolver/0.log" Sep 10 19:20:56.207693 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.207656 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:20:56.207893 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:56.207772 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:20:56.207893 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:56.207831 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls podName:974975b6-d004-4708-a3d3-1757891d57a0 nodeName:}" failed. No retries permitted until 2026-09-10 19:21:04.207812409 +0000 UTC m=+50.175624634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls") pod "dns-default-klkw2" (UID: "974975b6-d004-4708-a3d3-1757891d57a0") : secret "dns-default-metrics-tls" not found Sep 10 19:20:56.750720 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.750696 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jzlhg_2ed20544-9b54-4e94-9ebb-7cf426aabff9/node-ca/0.log" Sep 10 19:20:56.762172 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.762145 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v" event={"ID":"9eab30f3-754a-47c2-a5e4-1c1e0faab235","Type":"ContainerStarted","Data":"13c0ba9e03ce04eab124baa38d57c7c7062d22ac01124625652f34a107f7906e"} Sep 10 19:20:56.907663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.907587 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-m5rb9"] Sep 10 19:20:56.932174 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.932143 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-m5rb9"] Sep 10 19:20:56.932342 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.932239 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:56.934698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.934674 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 10 19:20:56.934698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.934684 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lktbl\"" Sep 10 19:20:56.935427 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.935403 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 10 19:20:56.935549 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.935412 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 10 19:20:56.935549 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:56.935469 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 10 19:20:57.014587 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.014553 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/521b17a7-8541-4f4c-adde-6f5bf584c707-signing-cabundle\") pod \"service-ca-75d99c484-m5rb9\" (UID: \"521b17a7-8541-4f4c-adde-6f5bf584c707\") " pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:57.014796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.014671 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh9xg\" (UniqueName: \"kubernetes.io/projected/521b17a7-8541-4f4c-adde-6f5bf584c707-kube-api-access-vh9xg\") pod \"service-ca-75d99c484-m5rb9\" (UID: \"521b17a7-8541-4f4c-adde-6f5bf584c707\") " pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:57.014796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.014708 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/521b17a7-8541-4f4c-adde-6f5bf584c707-signing-key\") pod \"service-ca-75d99c484-m5rb9\" (UID: \"521b17a7-8541-4f4c-adde-6f5bf584c707\") " pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:57.116062 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.116022 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh9xg\" (UniqueName: \"kubernetes.io/projected/521b17a7-8541-4f4c-adde-6f5bf584c707-kube-api-access-vh9xg\") pod \"service-ca-75d99c484-m5rb9\" (UID: \"521b17a7-8541-4f4c-adde-6f5bf584c707\") " pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:57.116062 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.116067 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/521b17a7-8541-4f4c-adde-6f5bf584c707-signing-key\") pod \"service-ca-75d99c484-m5rb9\" (UID: \"521b17a7-8541-4f4c-adde-6f5bf584c707\") " pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:57.116343 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.116258 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/521b17a7-8541-4f4c-adde-6f5bf584c707-signing-cabundle\") pod \"service-ca-75d99c484-m5rb9\" (UID: \"521b17a7-8541-4f4c-adde-6f5bf584c707\") " pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:57.116973 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.116954 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/521b17a7-8541-4f4c-adde-6f5bf584c707-signing-cabundle\") pod \"service-ca-75d99c484-m5rb9\" (UID: \"521b17a7-8541-4f4c-adde-6f5bf584c707\") " pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:57.118525 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.118506 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/521b17a7-8541-4f4c-adde-6f5bf584c707-signing-key\") pod \"service-ca-75d99c484-m5rb9\" (UID: \"521b17a7-8541-4f4c-adde-6f5bf584c707\") " pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:57.124559 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.124534 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh9xg\" (UniqueName: \"kubernetes.io/projected/521b17a7-8541-4f4c-adde-6f5bf584c707-kube-api-access-vh9xg\") pod \"service-ca-75d99c484-m5rb9\" (UID: \"521b17a7-8541-4f4c-adde-6f5bf584c707\") " pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:57.241161 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.241071 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-m5rb9" Sep 10 19:20:57.317837 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.317802 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:57.318059 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.317857 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:20:57.318059 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:57.317932 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:20:57.318059 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:57.317986 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm podName:56353bf4-bd0a-4710-9f0e-35d21ddda90b nodeName:}" failed. No retries permitted until 2026-09-10 19:21:01.317971587 +0000 UTC m=+47.285783817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzjrz" (UID: "56353bf4-bd0a-4710-9f0e-35d21ddda90b") : configmap "kube-rbac-proxy" not found Sep 10 19:20:57.318059 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:57.317980 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:20:57.318059 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:20:57.318019 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls podName:56353bf4-bd0a-4710-9f0e-35d21ddda90b nodeName:}" failed. No retries permitted until 2026-09-10 19:21:01.318010953 +0000 UTC m=+47.285823179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjrz" (UID: "56353bf4-bd0a-4710-9f0e-35d21ddda90b") : secret "insights-runtime-extractor-tls" not found Sep 10 19:20:57.546191 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.546115 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-m5rb9"] Sep 10 19:20:57.549455 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:20:57.549426 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521b17a7_8541_4f4c_adde_6f5bf584c707.slice/crio-1fe7b5b3999ffb589d37ac6e4a9a7c8390512cd56c52bdeb9b80cd6ec1a3ee62 WatchSource:0}: Error finding container 1fe7b5b3999ffb589d37ac6e4a9a7c8390512cd56c52bdeb9b80cd6ec1a3ee62: Status 404 returned error can't find the container with id 1fe7b5b3999ffb589d37ac6e4a9a7c8390512cd56c52bdeb9b80cd6ec1a3ee62 Sep 10 19:20:57.550583 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.550547 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wv5m9_23f79489-278b-41b0-b002-474e6da0dc7c/serve-healthcheck-canary/0.log" Sep 10 19:20:57.766922 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.766881 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v" event={"ID":"9eab30f3-754a-47c2-a5e4-1c1e0faab235","Type":"ContainerStarted","Data":"d93edd5325857025f36381ae01d3c4ed1865e00c57ce2d3eb3fc99c037b559c4"} Sep 10 19:20:57.768054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.768024 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-m5rb9" event={"ID":"521b17a7-8541-4f4c-adde-6f5bf584c707","Type":"ContainerStarted","Data":"1fe7b5b3999ffb589d37ac6e4a9a7c8390512cd56c52bdeb9b80cd6ec1a3ee62"} Sep 10 19:20:57.783656 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:20:57.783603 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-d845v" podStartSLOduration=1.906115615 podStartE2EDuration="3.78358414s" totalImagesPullingTime="1.877468525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:57.783020206 +0000 UTC m=+43.750832494" watchObservedRunningTime="2026-09-10 19:20:57.78358414 +0000 UTC m=+43.751396381" Sep 10 19:21:00.778221 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:00.778178 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-m5rb9" event={"ID":"521b17a7-8541-4f4c-adde-6f5bf584c707","Type":"ContainerStarted","Data":"c5749b3ffedafb6665467325a3545540f79e55fb60e981e74d510fa29c0633d5"} Sep 10 19:21:01.352211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:01.352170 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:21:01.352445 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:01.352230 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:21:01.352445 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:21:01.352344 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:21:01.352445 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:21:01.352414 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls podName:56353bf4-bd0a-4710-9f0e-35d21ddda90b nodeName:}" failed. No retries permitted until 2026-09-10 19:21:09.352398197 +0000 UTC m=+55.320210421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjrz" (UID: "56353bf4-bd0a-4710-9f0e-35d21ddda90b") : secret "insights-runtime-extractor-tls" not found Sep 10 19:21:01.352927 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:01.352904 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56353bf4-bd0a-4710-9f0e-35d21ddda90b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:21:04.275250 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:04.275210 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:21:04.275697 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:21:04.275342 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:21:04.275697 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:21:04.275397 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls podName:974975b6-d004-4708-a3d3-1757891d57a0 nodeName:}" failed. No retries permitted until 2026-09-10 19:21:20.275382572 +0000 UTC m=+66.243194797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls") pod "dns-default-klkw2" (UID: "974975b6-d004-4708-a3d3-1757891d57a0") : secret "dns-default-metrics-tls" not found Sep 10 19:21:09.413894 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:09.413850 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:21:09.416220 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:09.416196 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56353bf4-bd0a-4710-9f0e-35d21ddda90b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjrz\" (UID: \"56353bf4-bd0a-4710-9f0e-35d21ddda90b\") " pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:21:09.516433 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:09.516394 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzjrz" Sep 10 19:21:09.790017 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:09.789886 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-m5rb9" podStartSLOduration=11.489638971 podStartE2EDuration="13.789862436s" totalImagesPullingTime="2.300223465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:00.800199426 +0000 UTC m=+46.768011674" watchObservedRunningTime="2026-09-10 19:21:09.789862436 +0000 UTC m=+55.757674687" Sep 10 19:21:09.790247 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:09.790227 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzjrz"] Sep 10 19:21:09.793943 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:21:09.793910 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56353bf4_bd0a_4710_9f0e_35d21ddda90b.slice/crio-303084f9fc1a58ffb637fdbde7353979ab1adda5fd8d317c57cea1c0b38cc220 WatchSource:0}: Error finding container 303084f9fc1a58ffb637fdbde7353979ab1adda5fd8d317c57cea1c0b38cc220: Status 404 returned error can't find the container with id 303084f9fc1a58ffb637fdbde7353979ab1adda5fd8d317c57cea1c0b38cc220 Sep 10 19:21:09.802183 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:09.802159 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzjrz" event={"ID":"56353bf4-bd0a-4710-9f0e-35d21ddda90b","Type":"ContainerStarted","Data":"303084f9fc1a58ffb637fdbde7353979ab1adda5fd8d317c57cea1c0b38cc220"} Sep 10 19:21:10.807311 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:10.807258 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzjrz" event={"ID":"56353bf4-bd0a-4710-9f0e-35d21ddda90b","Type":"ContainerStarted","Data":"fe4023552cde943720e697d45adbf794794567cf33bb6986320ac6b02ca80e75"} Sep 10 19:21:10.807706 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:10.807334 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzjrz" event={"ID":"56353bf4-bd0a-4710-9f0e-35d21ddda90b","Type":"ContainerStarted","Data":"f25f0e26b38d88735a00510053ca2730046bbab325e602aeca6d94137e1979da"} Sep 10 19:21:12.816880 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:12.816842 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzjrz" event={"ID":"56353bf4-bd0a-4710-9f0e-35d21ddda90b","Type":"ContainerStarted","Data":"8d3bd430040b3a6912f0fe148d637d844e151417601d03b8e26386f10f4a5917"} Sep 10 19:21:12.838245 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:12.838152 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hzjrz" podStartSLOduration=17.626691561 podStartE2EDuration="19.838134247s" totalImagesPullingTime="2.211442686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:12.837468408 +0000 UTC m=+58.805280659" watchObservedRunningTime="2026-09-10 19:21:12.838134247 +0000 UTC m=+58.805946494" Sep 10 19:21:14.723017 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:14.722987 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9q29" Sep 10 19:21:20.191122 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.191080 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:21:20.193844 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.193818 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 19:21:20.203855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.203822 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e94806-d52b-476f-bf6b-11fab94483a6-metrics-certs\") pod \"network-metrics-daemon-4fqpp\" (UID: \"29e94806-d52b-476f-bf6b-11fab94483a6\") " pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:21:20.291903 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.291855 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwblm\" (UniqueName: \"kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm\") pod \"network-check-target-gwpjk\" (UID: \"71bc58ba-c381-420f-aa47-c298bda76828\") " pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:21:20.292107 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.291920 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:21:20.292107 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.291954 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:21:20.294251 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.294221 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 19:21:20.294405 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.294319 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 19:21:20.294521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.294500 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/974975b6-d004-4708-a3d3-1757891d57a0-metrics-tls\") pod \"dns-default-klkw2\" (UID: \"974975b6-d004-4708-a3d3-1757891d57a0\") " pod="openshift-dns/dns-default-klkw2" Sep 10 19:21:20.296252 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.296231 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npjmj\"" Sep 10 19:21:20.304077 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.304028 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4fqpp" Sep 10 19:21:20.304720 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.304699 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 19:21:20.305052 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.305032 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dae04ef-d3ba-4c10-8fc6-9eef8093c500-original-pull-secret\") pod \"global-pull-secret-syncer-jclqb\" (UID: \"9dae04ef-d3ba-4c10-8fc6-9eef8093c500\") " pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:21:20.315788 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.315762 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwblm\" (UniqueName: \"kubernetes.io/projected/71bc58ba-c381-420f-aa47-c298bda76828-kube-api-access-xwblm\") pod \"network-check-target-gwpjk\" (UID: \"71bc58ba-c381-420f-aa47-c298bda76828\") " pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:21:20.395471 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.395219 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-plphg\"" Sep 10 19:21:20.403600 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.403319 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:21:20.509754 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.509726 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2kts8\"" Sep 10 19:21:20.518657 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.518632 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-klkw2" Sep 10 19:21:20.599821 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.599790 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jclqb" Sep 10 19:21:20.621173 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.621144 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4fqpp"] Sep 10 19:21:20.626456 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:21:20.626429 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e94806_d52b_476f_bf6b_11fab94483a6.slice/crio-a0d91bc213cf2fc63d6431df943520821673a439aa68b08aa189a478a9b0514b WatchSource:0}: Error finding container a0d91bc213cf2fc63d6431df943520821673a439aa68b08aa189a478a9b0514b: Status 404 returned error can't find the container with id a0d91bc213cf2fc63d6431df943520821673a439aa68b08aa189a478a9b0514b Sep 10 19:21:20.739873 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.739798 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gwpjk"] Sep 10 19:21:20.743178 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:21:20.743149 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bc58ba_c381_420f_aa47_c298bda76828.slice/crio-24d173de51d2e27ed5615fd411486a9fd22ddb59d7e8a4a9a0230630e4fe890e WatchSource:0}: Error finding container 24d173de51d2e27ed5615fd411486a9fd22ddb59d7e8a4a9a0230630e4fe890e: Status 404 returned error can't find the container with id 24d173de51d2e27ed5615fd411486a9fd22ddb59d7e8a4a9a0230630e4fe890e Sep 10 19:21:20.773681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.773648 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf"] Sep 10 19:21:20.801412 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.801379 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.801776 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.801749 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-ca\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.801776 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.801782 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.801983 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.801852 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-hub\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.801983 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.801887 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.801983 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.801906 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ddl\" (UniqueName: \"kubernetes.io/projected/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-kube-api-access-d4ddl\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.801983 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.801958 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.808831 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.808594 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 10 19:21:20.808831 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.808618 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 19:21:20.809241 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.809220 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 10 19:21:20.809363 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.809220 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 19:21:20.809528 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.809514 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 19:21:20.809670 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.809616 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf"] Sep 10 19:21:20.812617 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.812430 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 10 19:21:20.812617 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.812490 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 10 19:21:20.842662 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.842632 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gwpjk" event={"ID":"71bc58ba-c381-420f-aa47-c298bda76828","Type":"ContainerStarted","Data":"24d173de51d2e27ed5615fd411486a9fd22ddb59d7e8a4a9a0230630e4fe890e"} Sep 10 19:21:20.843745 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.843718 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4fqpp" event={"ID":"29e94806-d52b-476f-bf6b-11fab94483a6","Type":"ContainerStarted","Data":"a0d91bc213cf2fc63d6431df943520821673a439aa68b08aa189a478a9b0514b"} Sep 10 19:21:20.847042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.847019 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-klkw2"] Sep 10 19:21:20.851585 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:21:20.851555 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974975b6_d004_4708_a3d3_1757891d57a0.slice/crio-bcd581f3ebba64b36cc31face997de303e9786c433aed54adbef5a7a3058d29f WatchSource:0}: Error finding container bcd581f3ebba64b36cc31face997de303e9786c433aed54adbef5a7a3058d29f: Status 404 returned error can't find the container with id bcd581f3ebba64b36cc31face997de303e9786c433aed54adbef5a7a3058d29f Sep 10 19:21:20.902467 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.902425 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.902651 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.902475 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-ca\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.902651 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.902495 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.902651 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.902523 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-hub\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.902651 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.902546 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.902651 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.902562 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ddl\" (UniqueName: \"kubernetes.io/projected/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-kube-api-access-d4ddl\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.903227 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.903194 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.905224 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.905197 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-hub\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.905375 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.905327 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.905590 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.905568 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-ca\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.905627 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.905579 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.931063 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.931032 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ddl\" (UniqueName: \"kubernetes.io/projected/bb8d1ac0-6f81-4ca6-8dd8-df400ce51025-kube-api-access-d4ddl\") pod \"cluster-proxy-proxy-agent-6d97bfcc96-4krhf\" (UID: \"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:20.950185 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:20.950154 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jclqb"] Sep 10 19:21:20.953344 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:21:20.953314 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dae04ef_d3ba_4c10_8fc6_9eef8093c500.slice/crio-cb82ee0fbab96cbde1dabd83e6c3bcaa2bb9c449b743aa1762e19f49366f9a05 WatchSource:0}: Error finding container cb82ee0fbab96cbde1dabd83e6c3bcaa2bb9c449b743aa1762e19f49366f9a05: Status 404 returned error can't find the container with id cb82ee0fbab96cbde1dabd83e6c3bcaa2bb9c449b743aa1762e19f49366f9a05 Sep 10 19:21:21.120818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:21.120778 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" Sep 10 19:21:21.424918 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:21.424832 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf"] Sep 10 19:21:21.429409 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:21:21.429374 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb8d1ac0_6f81_4ca6_8dd8_df400ce51025.slice/crio-7ae6d42e5c69a6580137aaa9498ebdc4f89e7c60c79957f6fb3209df24346b9b WatchSource:0}: Error finding container 7ae6d42e5c69a6580137aaa9498ebdc4f89e7c60c79957f6fb3209df24346b9b: Status 404 returned error can't find the container with id 7ae6d42e5c69a6580137aaa9498ebdc4f89e7c60c79957f6fb3209df24346b9b Sep 10 19:21:21.851666 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:21.851619 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jclqb" event={"ID":"9dae04ef-d3ba-4c10-8fc6-9eef8093c500","Type":"ContainerStarted","Data":"cb82ee0fbab96cbde1dabd83e6c3bcaa2bb9c449b743aa1762e19f49366f9a05"} Sep 10 19:21:21.853908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:21.853835 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" event={"ID":"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025","Type":"ContainerStarted","Data":"7ae6d42e5c69a6580137aaa9498ebdc4f89e7c60c79957f6fb3209df24346b9b"} Sep 10 19:21:21.855451 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:21.855374 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-klkw2" event={"ID":"974975b6-d004-4708-a3d3-1757891d57a0","Type":"ContainerStarted","Data":"bcd581f3ebba64b36cc31face997de303e9786c433aed54adbef5a7a3058d29f"} Sep 10 19:21:28.558622 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.558526 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66cc6c9567-gt4hh"] Sep 10 19:21:28.561172 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.561148 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.563596 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.563572 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 19:21:28.564489 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.564434 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 19:21:28.564489 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.564478 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c5tbm\"" Sep 10 19:21:28.564667 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.564517 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 19:21:28.564667 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.564588 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 19:21:28.564779 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.564719 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 19:21:28.564867 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.564847 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 19:21:28.564982 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.564891 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 19:21:28.569061 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.569041 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 19:21:28.576742 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.576719 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66cc6c9567-gt4hh"] Sep 10 19:21:28.662229 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.662188 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-oauth-config\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.662229 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.662233 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jxz\" (UniqueName: \"kubernetes.io/projected/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-kube-api-access-s9jxz\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.662543 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.662256 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-service-ca\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.662543 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.662343 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-serving-cert\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.662543 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.662423 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-trusted-ca-bundle\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.662543 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.662449 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-config\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.662543 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.662482 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-oauth-serving-cert\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.763616 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.763570 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jxz\" (UniqueName: \"kubernetes.io/projected/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-kube-api-access-s9jxz\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.763616 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.763617 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-service-ca\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.763869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.763634 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-serving-cert\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.763869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.763775 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-trusted-ca-bundle\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.763869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.763812 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-config\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.763869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.763830 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-oauth-serving-cert\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.763869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.763867 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-oauth-config\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.764471 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.764445 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-service-ca\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.764611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.764566 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-config\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.764700 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.764674 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-trusted-ca-bundle\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.764801 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.764779 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-oauth-serving-cert\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.766201 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.766178 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-serving-cert\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.766201 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.766199 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-oauth-config\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.772317 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.772296 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jxz\" (UniqueName: \"kubernetes.io/projected/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-kube-api-access-s9jxz\") pod \"console-66cc6c9567-gt4hh\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.870785 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.870700 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:28.886706 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.886673 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" event={"ID":"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025","Type":"ContainerStarted","Data":"614a68749a344e5fede2bbc3b3f4523b8baa7410f16deb628dd97065ce4f618e"} Sep 10 19:21:28.886826 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.886721 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" event={"ID":"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025","Type":"ContainerStarted","Data":"a597250bf1a988c6c8053a33a53fb62b8235647e1f45c169f22fe920f29d90dd"} Sep 10 19:21:28.886826 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.886734 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" event={"ID":"bb8d1ac0-6f81-4ca6-8dd8-df400ce51025","Type":"ContainerStarted","Data":"f073e67d3913235421e2c603d820c6fa7919fdf377189ac4eb4eadda3907c1bf"} Sep 10 19:21:28.888602 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.888577 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-klkw2" event={"ID":"974975b6-d004-4708-a3d3-1757891d57a0","Type":"ContainerStarted","Data":"d983dd712429fe80ac0d0b76960af993328e16ecb4b5ee7832465cc6791ed8f4"} Sep 10 19:21:28.888700 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.888666 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-klkw2" event={"ID":"974975b6-d004-4708-a3d3-1757891d57a0","Type":"ContainerStarted","Data":"3b50a517dc15e16acfe6a11ff228d13f80b7317c59d8c0a689c614908337a066"} Sep 10 19:21:28.888770 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.888699 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-klkw2" Sep 10 19:21:28.890117 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.890092 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gwpjk" event={"ID":"71bc58ba-c381-420f-aa47-c298bda76828","Type":"ContainerStarted","Data":"1ee490489f7bfd16a60e6b44f66d93c01ae2f2d7df71963e6d99f9e9871cf147"} Sep 10 19:21:28.890197 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.890149 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:21:28.892249 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.892166 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4fqpp" event={"ID":"29e94806-d52b-476f-bf6b-11fab94483a6","Type":"ContainerStarted","Data":"5d0901d9fd9411fedb1448fb390e6ffdc3915adc104d61c68e5c741485036d99"} Sep 10 19:21:28.892249 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.892215 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4fqpp" event={"ID":"29e94806-d52b-476f-bf6b-11fab94483a6","Type":"ContainerStarted","Data":"ff5c1f8254ddd8b44405650356b276643488e82001219cb6e7008494fd38af15"} Sep 10 19:21:28.893795 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.893775 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jclqb" event={"ID":"9dae04ef-d3ba-4c10-8fc6-9eef8093c500","Type":"ContainerStarted","Data":"39c96e08b36c7e517672017800efeeb6250b503db557ee2038fd9f49dbb63a06"} Sep 10 19:21:28.910908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.910855 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d97bfcc96-4krhf" podStartSLOduration=2.257242356 podStartE2EDuration="8.910838278s" totalImagesPullingTime="6.653595922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:28.910020019 +0000 UTC m=+74.877832265" watchObservedRunningTime="2026-09-10 19:21:28.910838278 +0000 UTC m=+74.878650578" Sep 10 19:21:28.936016 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.935830 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jclqb" podStartSLOduration=66.814004778 podStartE2EDuration="1m13.93579985s" totalImagesPullingTime="7.121795072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:28.934762857 +0000 UTC m=+74.902575107" watchObservedRunningTime="2026-09-10 19:21:28.93579985 +0000 UTC m=+74.903612098" Sep 10 19:21:28.955419 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.955364 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4fqpp" podStartSLOduration=67.523808892 podStartE2EDuration="1m14.955345913s" totalImagesPullingTime="7.431537021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:28.955001671 +0000 UTC m=+74.922813915" watchObservedRunningTime="2026-09-10 19:21:28.955345913 +0000 UTC m=+74.923158168" Sep 10 19:21:28.979251 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.979196 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-klkw2" podStartSLOduration=33.767984388 podStartE2EDuration="40.979176519s" totalImagesPullingTime="7.211192131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:28.977905781 +0000 UTC m=+74.945718047" watchObservedRunningTime="2026-09-10 19:21:28.979176519 +0000 UTC m=+74.946988766" Sep 10 19:21:28.994102 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:28.994048 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gwpjk" podStartSLOduration=66.674284 podStartE2EDuration="1m13.994029668s" totalImagesPullingTime="7.319745668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:28.993008553 +0000 UTC m=+74.960820800" watchObservedRunningTime="2026-09-10 19:21:28.994029668 +0000 UTC m=+74.961841914" Sep 10 19:21:29.143737 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:29.143653 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66cc6c9567-gt4hh"] Sep 10 19:21:29.146594 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:21:29.146564 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbea0b8_e383_4b97_9b15_1b1ad2a8edc8.slice/crio-6109a607168942113c0ae7aefedc849eb24fa696cbb875f3d56be638d5ca944b WatchSource:0}: Error finding container 6109a607168942113c0ae7aefedc849eb24fa696cbb875f3d56be638d5ca944b: Status 404 returned error can't find the container with id 6109a607168942113c0ae7aefedc849eb24fa696cbb875f3d56be638d5ca944b Sep 10 19:21:29.898892 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:29.898847 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cc6c9567-gt4hh" event={"ID":"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8","Type":"ContainerStarted","Data":"6109a607168942113c0ae7aefedc849eb24fa696cbb875f3d56be638d5ca944b"} Sep 10 19:21:32.911121 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:32.911081 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cc6c9567-gt4hh" event={"ID":"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8","Type":"ContainerStarted","Data":"11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd"} Sep 10 19:21:32.931546 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:32.931480 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66cc6c9567-gt4hh" podStartSLOduration=2.218671108 podStartE2EDuration="4.931462815s" totalImagesPullingTime="2.712791707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:32.930567637 +0000 UTC m=+78.898379877" watchObservedRunningTime="2026-09-10 19:21:32.931462815 +0000 UTC m=+78.899275063" Sep 10 19:21:33.565841 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.565796 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kh7qh"] Sep 10 19:21:33.568588 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.568568 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.571063 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.571029 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 19:21:33.571992 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.571965 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 19:21:33.572103 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.572080 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 19:21:33.572220 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.572111 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 19:21:33.572220 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.571997 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dvs6s\"" Sep 10 19:21:33.572220 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.572031 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 19:21:33.572395 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.571997 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 19:21:33.601663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.601627 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-sys\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.601828 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.601676 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-wtmp\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.601828 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.601704 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-metrics-client-ca\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.601828 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.601754 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbx7d\" (UniqueName: \"kubernetes.io/projected/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-kube-api-access-cbx7d\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.601828 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.601781 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.601986 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.601859 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.601986 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.601914 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-root\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.601986 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.601944 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-tls\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.601986 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.601974 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-textfile\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703259 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703219 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-root\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703259 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703261 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-tls\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703298 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-textfile\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703327 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-sys\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703361 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-wtmp\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703369 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-root\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703383 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-metrics-client-ca\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703527 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:21:33.703427 2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 19:21:33.703527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703444 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbx7d\" (UniqueName: \"kubernetes.io/projected/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-kube-api-access-cbx7d\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703458 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-sys\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703480 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.703527 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:21:33.703508 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-tls podName:615528aa-2d5f-47fe-80a4-5d83fbeac9a3 nodeName:}" failed. No retries permitted until 2026-09-10 19:21:34.203485036 +0000 UTC m=+80.171297269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-tls") pod "node-exporter-kh7qh" (UID: "615528aa-2d5f-47fe-80a4-5d83fbeac9a3") : secret "node-exporter-tls" not found Sep 10 19:21:33.704023 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703612 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-wtmp\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.704023 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703677 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.704023 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703698 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-textfile\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.704023 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.703914 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-metrics-client-ca\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.704271 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.704250 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.705973 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.705954 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:33.712064 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:33.712039 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbx7d\" (UniqueName: \"kubernetes.io/projected/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-kube-api-access-cbx7d\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:34.206656 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:34.206599 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-tls\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:34.208949 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:34.208923 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/615528aa-2d5f-47fe-80a4-5d83fbeac9a3-node-exporter-tls\") pod \"node-exporter-kh7qh\" (UID: \"615528aa-2d5f-47fe-80a4-5d83fbeac9a3\") " pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:34.479218 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:34.479125 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kh7qh" Sep 10 19:21:34.492238 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:21:34.492204 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615528aa_2d5f_47fe_80a4_5d83fbeac9a3.slice/crio-64dc7debd45bc7e721fbb583e2068effa65031aa8ad3a9f2476b5c36b05a004b WatchSource:0}: Error finding container 64dc7debd45bc7e721fbb583e2068effa65031aa8ad3a9f2476b5c36b05a004b: Status 404 returned error can't find the container with id 64dc7debd45bc7e721fbb583e2068effa65031aa8ad3a9f2476b5c36b05a004b Sep 10 19:21:34.920309 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:34.920256 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kh7qh" event={"ID":"615528aa-2d5f-47fe-80a4-5d83fbeac9a3","Type":"ContainerStarted","Data":"64dc7debd45bc7e721fbb583e2068effa65031aa8ad3a9f2476b5c36b05a004b"} Sep 10 19:21:35.536142 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.536110 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-687966b585-xd8ng"] Sep 10 19:21:35.543644 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.543622 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.546691 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.546669 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 19:21:35.547007 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.546985 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 19:21:35.547146 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.547049 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 19:21:35.547681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.547542 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 19:21:35.547681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.547556 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6547k\"" Sep 10 19:21:35.547681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.547573 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 19:21:35.547681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.547574 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7qk7kqkmp6hlf\"" Sep 10 19:21:35.553595 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.553570 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-687966b585-xd8ng"] Sep 10 19:21:35.617904 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.617866 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-metrics-client-ca\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.617904 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.617909 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k29ls\" (UniqueName: \"kubernetes.io/projected/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-kube-api-access-k29ls\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.618132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.617936 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.618132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.618061 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-tls\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.618132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.618106 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.618255 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.618134 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.618255 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.618171 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.618255 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.618197 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-grpc-tls\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.719427 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.719382 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.719599 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.719441 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-tls\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.719599 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.719470 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.719599 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.719495 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.719599 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.719536 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.719802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.719673 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-grpc-tls\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.719802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.719728 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-metrics-client-ca\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.719802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.719759 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k29ls\" (UniqueName: \"kubernetes.io/projected/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-kube-api-access-k29ls\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.720990 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.720949 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-metrics-client-ca\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.722250 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.722213 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.722417 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.722395 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-tls\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.722526 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.722506 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.722646 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.722627 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.722691 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.722644 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.722729 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.722715 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-secret-grpc-tls\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.729176 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.729152 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k29ls\" (UniqueName: \"kubernetes.io/projected/ab0cc04f-dee0-45e2-8e78-ed57c3d60cce-kube-api-access-k29ls\") pod \"thanos-querier-687966b585-xd8ng\" (UID: \"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce\") " pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.853110 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.853073 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:35.925763 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.925729 2584 generic.go:356] "Generic (PLEG): container finished" podID="615528aa-2d5f-47fe-80a4-5d83fbeac9a3" containerID="3c98e66a9e284234d0c9f0f63413e8e4bfb0e5d354f6fb7e7db54c474031ba41" exitCode=0 Sep 10 19:21:35.925891 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:35.925772 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kh7qh" event={"ID":"615528aa-2d5f-47fe-80a4-5d83fbeac9a3","Type":"ContainerDied","Data":"3c98e66a9e284234d0c9f0f63413e8e4bfb0e5d354f6fb7e7db54c474031ba41"} Sep 10 19:21:36.132087 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:36.131998 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-687966b585-xd8ng"] Sep 10 19:21:36.135491 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:21:36.135449 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0cc04f_dee0_45e2_8e78_ed57c3d60cce.slice/crio-8f4f9270a310111468f0de55e455316f973b98ad8f2820685101a609c41574c2 WatchSource:0}: Error finding container 8f4f9270a310111468f0de55e455316f973b98ad8f2820685101a609c41574c2: Status 404 returned error can't find the container with id 8f4f9270a310111468f0de55e455316f973b98ad8f2820685101a609c41574c2 Sep 10 19:21:36.932714 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:36.932436 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kh7qh" event={"ID":"615528aa-2d5f-47fe-80a4-5d83fbeac9a3","Type":"ContainerStarted","Data":"54397efca5d8af5d174a6c16c72e68a935de9914c12d9903bda5aaf0c9af841b"} Sep 10 19:21:36.932714 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:36.932500 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kh7qh" event={"ID":"615528aa-2d5f-47fe-80a4-5d83fbeac9a3","Type":"ContainerStarted","Data":"f85cd604fd59d6948dbbd86e5e8321d7a427c8fb7563b140aa600e2a8d420a42"} Sep 10 19:21:36.934005 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:36.933978 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" event={"ID":"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce","Type":"ContainerStarted","Data":"8f4f9270a310111468f0de55e455316f973b98ad8f2820685101a609c41574c2"} Sep 10 19:21:38.871161 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:38.871116 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:38.871661 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:38.871244 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:38.875960 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:38.875933 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:38.901376 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:38.901320 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kh7qh" podStartSLOduration=4.840671235 podStartE2EDuration="5.901269492s" totalImagesPullingTime="996.836266ms" totalInitContainerRuntime="63.761991ms" isStatefulPod=true podCreationTimestamp="2026-09-10 19:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:36.952862558 +0000 UTC m=+82.920674821" watchObservedRunningTime="2026-09-10 19:21:38.901269492 +0000 UTC m=+84.869081741" Sep 10 19:21:38.901666 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:38.901646 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-klkw2" Sep 10 19:21:38.945134 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:38.945088 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" event={"ID":"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce","Type":"ContainerStarted","Data":"f8d212a01bb95a39dc635699de3dc39191a76ae125096c61016617710f9da007"} Sep 10 19:21:38.945358 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:38.945166 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" event={"ID":"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce","Type":"ContainerStarted","Data":"1a6556a4d1a962100616dcad067d3a711fd338dbd953ebf05de687758e1e1eec"} Sep 10 19:21:38.945358 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:38.945195 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" event={"ID":"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce","Type":"ContainerStarted","Data":"d3207e54b6312972d43ce0e8a1019b8e4e420f48de88cd02e1b6643c2e968ef4"} Sep 10 19:21:38.949405 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:38.949376 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:21:39.952246 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:39.952210 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" event={"ID":"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce","Type":"ContainerStarted","Data":"5ca005b330ab68bfe976d5ab15f2deddb8e787030f62aea0e6605f5a1dc61af3"} Sep 10 19:21:39.952741 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:39.952314 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" event={"ID":"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce","Type":"ContainerStarted","Data":"6f0e0fd507305759625997e957968e3d4c21f5b16c48ef9932d22dc7eb936c72"} Sep 10 19:21:39.952741 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:39.952358 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" event={"ID":"ab0cc04f-dee0-45e2-8e78-ed57c3d60cce","Type":"ContainerStarted","Data":"810a4cc38c3e5ef2f0bb4d7d3b124cb35d2c6b060eee9c0f9a598d64fea17b7b"} Sep 10 19:21:40.017571 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:40.017457 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" podStartSLOduration=1.6195724500000002 podStartE2EDuration="5.017438253s" totalImagesPullingTime="3.397865803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:21:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:40.015375115 +0000 UTC m=+85.983187363" watchObservedRunningTime="2026-09-10 19:21:40.017438253 +0000 UTC m=+85.985250500" Sep 10 19:21:40.955708 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:40.955677 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:46.965572 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:46.965539 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-687966b585-xd8ng" Sep 10 19:21:52.417669 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:52.417631 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66cc6c9567-gt4hh"] Sep 10 19:21:59.902161 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:21:59.902079 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gwpjk" Sep 10 19:22:17.441781 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.441709 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66cc6c9567-gt4hh" podUID="4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" containerName="console" containerID="cri-o://11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd" gracePeriod=15 Sep 10 19:22:17.688703 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.688674 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:22:17.836427 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.836392 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9jxz\" (UniqueName: \"kubernetes.io/projected/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-kube-api-access-s9jxz\") pod \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " Sep 10 19:22:17.836584 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.836439 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-service-ca\") pod \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " Sep 10 19:22:17.836584 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.836472 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-trusted-ca-bundle\") pod \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " Sep 10 19:22:17.836584 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.836494 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-oauth-serving-cert\") pod \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " Sep 10 19:22:17.836584 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.836548 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-oauth-config\") pod \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " Sep 10 19:22:17.836584 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.836581 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-serving-cert\") pod \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " Sep 10 19:22:17.836889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.836614 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-config\") pod \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\" (UID: \"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8\") " Sep 10 19:22:17.836946 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.836898 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-service-ca" (OuterVolumeSpecName: "service-ca") pod "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" (UID: "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:22:17.837037 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.837006 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-config" (OuterVolumeSpecName: "console-config") pod "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" (UID: "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:22:17.837085 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.837044 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" (UID: "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:22:17.837085 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.837065 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" (UID: "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:22:17.839014 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.838987 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" (UID: "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:22:17.839099 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.839008 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" (UID: "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:22:17.839099 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.838990 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-kube-api-access-s9jxz" (OuterVolumeSpecName: "kube-api-access-s9jxz") pod "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" (UID: "4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8"). InnerVolumeSpecName "kube-api-access-s9jxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:22:17.937557 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.937512 2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-config\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:22:17.937557 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.937546 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9jxz\" (UniqueName: \"kubernetes.io/projected/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-kube-api-access-s9jxz\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:22:17.937557 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.937556 2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-service-ca\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:22:17.937557 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.937565 2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-trusted-ca-bundle\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:22:17.937827 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.937575 2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-oauth-serving-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:22:17.937827 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.937584 2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-oauth-config\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:22:17.937827 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:17.937592 2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8-console-serving-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:22:18.091074 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:18.090984 2584 generic.go:356] "Generic (PLEG): container finished" podID="4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" containerID="11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd" exitCode=0 Sep 10 19:22:18.091074 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:18.091029 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cc6c9567-gt4hh" Sep 10 19:22:18.091320 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:18.091031 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cc6c9567-gt4hh" event={"ID":"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8","Type":"ContainerDied","Data":"11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd"} Sep 10 19:22:18.091320 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:18.091167 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cc6c9567-gt4hh" event={"ID":"4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8","Type":"ContainerDied","Data":"6109a607168942113c0ae7aefedc849eb24fa696cbb875f3d56be638d5ca944b"} Sep 10 19:22:18.091320 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:18.091204 2584 scope.go:117] "RemoveContainer" containerID="11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd" Sep 10 19:22:18.100047 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:18.100026 2584 scope.go:117] "RemoveContainer" containerID="11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd" Sep 10 19:22:18.100362 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:22:18.100345 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd\": container with ID starting with 11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd not found: ID does not exist" containerID="11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd" Sep 10 19:22:18.100428 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:18.100374 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd"} err="failed to get container status \"11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd\": rpc error: code = NotFound desc = could not find container \"11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd\": container with ID starting with 11154776d84b20d745738c6a0f6a35b1835771dac50b6bb6142b2b35ade55dbd not found: ID does not exist" Sep 10 19:22:18.115231 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:18.115201 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66cc6c9567-gt4hh"] Sep 10 19:22:18.118816 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:18.118791 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66cc6c9567-gt4hh"] Sep 10 19:22:18.486984 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:22:18.486881 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" path="/var/lib/kubelet/pods/4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8/volumes" Sep 10 19:23:01.615776 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.615741 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b59c58d5f-2bz7j"] Sep 10 19:23:01.616207 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.615987 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" containerName="console" Sep 10 19:23:01.616207 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.615997 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" containerName="console" Sep 10 19:23:01.616207 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.616046 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbea0b8-e383-4b97-9b15-1b1ad2a8edc8" containerName="console" Sep 10 19:23:01.619530 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.619512 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.623847 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.623816 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 19:23:01.623959 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.623867 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 19:23:01.624405 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.624386 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 19:23:01.624512 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.624387 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 19:23:01.624659 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.624645 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 19:23:01.625054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.624941 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 19:23:01.625054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.625003 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c5tbm\"" Sep 10 19:23:01.626333 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.625857 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 19:23:01.628369 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.628350 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 19:23:01.635382 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.635359 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b59c58d5f-2bz7j"] Sep 10 19:23:01.649566 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.649529 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-service-ca\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.649724 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.649573 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-trusted-ca-bundle\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.649724 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.649604 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-oauth-config\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.649724 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.649670 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-serving-cert\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.649842 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.649729 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-console-config\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.649842 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.649760 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-oauth-serving-cert\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.649842 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.649787 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxhs\" (UniqueName: \"kubernetes.io/projected/160cb959-927f-41be-998d-addcec346bc0-kube-api-access-wmxhs\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.750588 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.750538 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-trusted-ca-bundle\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.750796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.750597 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-oauth-config\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.750796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.750635 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-serving-cert\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.750796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.750659 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-console-config\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.750796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.750688 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-oauth-serving-cert\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.750994 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.750830 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxhs\" (UniqueName: \"kubernetes.io/projected/160cb959-927f-41be-998d-addcec346bc0-kube-api-access-wmxhs\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.750994 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.750874 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-service-ca\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.751502 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.751474 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-oauth-serving-cert\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.751502 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.751496 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-console-config\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.751652 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.751521 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-trusted-ca-bundle\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.751652 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.751571 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-service-ca\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.753266 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.753249 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-oauth-config\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.753376 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.753359 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-serving-cert\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.760462 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.760432 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxhs\" (UniqueName: \"kubernetes.io/projected/160cb959-927f-41be-998d-addcec346bc0-kube-api-access-wmxhs\") pod \"console-6b59c58d5f-2bz7j\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:01.929450 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:01.929352 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:02.205697 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:02.205610 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b59c58d5f-2bz7j"] Sep 10 19:23:02.208534 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:23:02.208506 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160cb959_927f_41be_998d_addcec346bc0.slice/crio-4cf7b5de204b07a127242d9072465a42236adfbcb7a04ebb22f33d1f93e098b1 WatchSource:0}: Error finding container 4cf7b5de204b07a127242d9072465a42236adfbcb7a04ebb22f33d1f93e098b1: Status 404 returned error can't find the container with id 4cf7b5de204b07a127242d9072465a42236adfbcb7a04ebb22f33d1f93e098b1 Sep 10 19:23:02.254473 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:02.254437 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b59c58d5f-2bz7j" event={"ID":"160cb959-927f-41be-998d-addcec346bc0","Type":"ContainerStarted","Data":"4cf7b5de204b07a127242d9072465a42236adfbcb7a04ebb22f33d1f93e098b1"} Sep 10 19:23:03.259439 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:03.259400 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b59c58d5f-2bz7j" event={"ID":"160cb959-927f-41be-998d-addcec346bc0","Type":"ContainerStarted","Data":"85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7"} Sep 10 19:23:03.279889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:03.279832 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b59c58d5f-2bz7j" podStartSLOduration=2.279814051 podStartE2EDuration="2.279814051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:23:03.278629473 +0000 UTC m=+169.246441720" watchObservedRunningTime="2026-09-10 19:23:03.279814051 +0000 UTC m=+169.247626300" Sep 10 19:23:11.930529 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:11.930419 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:11.930529 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:11.930490 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:11.935365 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:11.935339 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:23:12.296039 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:23:12.295953 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:24:09.827414 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:09.827371 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c8ff65d6d-m2hmx"] Sep 10 19:24:09.829682 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:09.829655 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:09.853664 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:09.853629 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c8ff65d6d-m2hmx"] Sep 10 19:24:09.946526 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:09.946488 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-oauth-config\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:09.946526 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:09.946542 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-serving-cert\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:09.946760 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:09.946559 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-config\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:09.946760 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:09.946584 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-service-ca\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:09.946760 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:09.946607 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxvb\" (UniqueName: \"kubernetes.io/projected/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-kube-api-access-vlxvb\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:09.946760 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:09.946648 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-oauth-serving-cert\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:09.946760 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:09.946693 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-trusted-ca-bundle\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.047869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.047827 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-oauth-serving-cert\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.047869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.047874 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-trusted-ca-bundle\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.048474 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.047893 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-oauth-config\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.048474 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.047931 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-serving-cert\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.048474 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.047953 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-config\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.048474 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.047986 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-service-ca\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.048474 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.048032 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxvb\" (UniqueName: \"kubernetes.io/projected/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-kube-api-access-vlxvb\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.048788 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.048766 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-service-ca\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.048849 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.048765 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-config\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.048849 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.048815 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-oauth-serving-cert\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.048947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.048922 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-trusted-ca-bundle\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.050650 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.050622 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-oauth-config\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.050751 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.050662 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-serving-cert\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.058740 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.058717 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxvb\" (UniqueName: \"kubernetes.io/projected/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-kube-api-access-vlxvb\") pod \"console-5c8ff65d6d-m2hmx\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.138840 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.138750 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:10.434398 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.434314 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c8ff65d6d-m2hmx"] Sep 10 19:24:10.437854 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:24:10.437825 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd77b55c_e8fe_4654_b2c0_59abae29ae6f.slice/crio-717de21fadbe97f17c03f347fafbb04888bbaea5fc3e11a7da4e1c9e5fe58114 WatchSource:0}: Error finding container 717de21fadbe97f17c03f347fafbb04888bbaea5fc3e11a7da4e1c9e5fe58114: Status 404 returned error can't find the container with id 717de21fadbe97f17c03f347fafbb04888bbaea5fc3e11a7da4e1c9e5fe58114 Sep 10 19:24:10.500186 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.500150 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8ff65d6d-m2hmx" event={"ID":"bd77b55c-e8fe-4654-b2c0-59abae29ae6f","Type":"ContainerStarted","Data":"51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db"} Sep 10 19:24:10.500186 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.500196 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8ff65d6d-m2hmx" event={"ID":"bd77b55c-e8fe-4654-b2c0-59abae29ae6f","Type":"ContainerStarted","Data":"717de21fadbe97f17c03f347fafbb04888bbaea5fc3e11a7da4e1c9e5fe58114"} Sep 10 19:24:10.530125 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:10.530071 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c8ff65d6d-m2hmx" podStartSLOduration=1.530053025 podStartE2EDuration="1.530053025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:24:10.529176798 +0000 UTC m=+236.496989046" watchObservedRunningTime="2026-09-10 19:24:10.530053025 +0000 UTC m=+236.497865268" Sep 10 19:24:20.139420 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:20.139376 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:20.139880 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:20.139443 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:20.144072 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:20.144048 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:20.536825 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:20.536746 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:24:20.587273 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:20.587240 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b59c58d5f-2bz7j"] Sep 10 19:24:45.607831 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.607764 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b59c58d5f-2bz7j" podUID="160cb959-927f-41be-998d-addcec346bc0" containerName="console" containerID="cri-o://85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7" gracePeriod=15 Sep 10 19:24:45.867856 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.867791 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:24:45.905843 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.905800 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-service-ca\") pod \"160cb959-927f-41be-998d-addcec346bc0\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " Sep 10 19:24:45.906021 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.905858 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-oauth-config\") pod \"160cb959-927f-41be-998d-addcec346bc0\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " Sep 10 19:24:45.906021 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.905886 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-oauth-serving-cert\") pod \"160cb959-927f-41be-998d-addcec346bc0\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " Sep 10 19:24:45.906021 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.905997 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-serving-cert\") pod \"160cb959-927f-41be-998d-addcec346bc0\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " Sep 10 19:24:45.906167 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.906087 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-trusted-ca-bundle\") pod \"160cb959-927f-41be-998d-addcec346bc0\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " Sep 10 19:24:45.906167 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.906113 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-console-config\") pod \"160cb959-927f-41be-998d-addcec346bc0\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " Sep 10 19:24:45.906167 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.906144 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmxhs\" (UniqueName: \"kubernetes.io/projected/160cb959-927f-41be-998d-addcec346bc0-kube-api-access-wmxhs\") pod \"160cb959-927f-41be-998d-addcec346bc0\" (UID: \"160cb959-927f-41be-998d-addcec346bc0\") " Sep 10 19:24:45.906338 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.906254 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "160cb959-927f-41be-998d-addcec346bc0" (UID: "160cb959-927f-41be-998d-addcec346bc0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:24:45.906462 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.906430 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-service-ca" (OuterVolumeSpecName: "service-ca") pod "160cb959-927f-41be-998d-addcec346bc0" (UID: "160cb959-927f-41be-998d-addcec346bc0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:24:45.906583 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.906477 2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-oauth-serving-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:24:45.906583 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.906556 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-console-config" (OuterVolumeSpecName: "console-config") pod "160cb959-927f-41be-998d-addcec346bc0" (UID: "160cb959-927f-41be-998d-addcec346bc0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:24:45.906669 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.906585 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "160cb959-927f-41be-998d-addcec346bc0" (UID: "160cb959-927f-41be-998d-addcec346bc0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:24:45.908132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.908100 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "160cb959-927f-41be-998d-addcec346bc0" (UID: "160cb959-927f-41be-998d-addcec346bc0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:24:45.908217 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.908151 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "160cb959-927f-41be-998d-addcec346bc0" (UID: "160cb959-927f-41be-998d-addcec346bc0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:24:45.908217 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:45.908196 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160cb959-927f-41be-998d-addcec346bc0-kube-api-access-wmxhs" (OuterVolumeSpecName: "kube-api-access-wmxhs") pod "160cb959-927f-41be-998d-addcec346bc0" (UID: "160cb959-927f-41be-998d-addcec346bc0"). InnerVolumeSpecName "kube-api-access-wmxhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:24:46.007042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.006999 2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-service-ca\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.007042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.007030 2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-oauth-config\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.007042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.007039 2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/160cb959-927f-41be-998d-addcec346bc0-console-serving-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.007042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.007049 2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-trusted-ca-bundle\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.007042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.007058 2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/160cb959-927f-41be-998d-addcec346bc0-console-config\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.007378 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.007066 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmxhs\" (UniqueName: \"kubernetes.io/projected/160cb959-927f-41be-998d-addcec346bc0-kube-api-access-wmxhs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.624739 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.624699 2584 generic.go:356] "Generic (PLEG): container finished" podID="160cb959-927f-41be-998d-addcec346bc0" containerID="85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7" exitCode=0 Sep 10 19:24:46.625124 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.624750 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b59c58d5f-2bz7j" Sep 10 19:24:46.625124 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.624750 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b59c58d5f-2bz7j" event={"ID":"160cb959-927f-41be-998d-addcec346bc0","Type":"ContainerDied","Data":"85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7"} Sep 10 19:24:46.625124 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.624867 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b59c58d5f-2bz7j" event={"ID":"160cb959-927f-41be-998d-addcec346bc0","Type":"ContainerDied","Data":"4cf7b5de204b07a127242d9072465a42236adfbcb7a04ebb22f33d1f93e098b1"} Sep 10 19:24:46.625124 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.624896 2584 scope.go:117] "RemoveContainer" containerID="85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7" Sep 10 19:24:46.632972 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.632954 2584 scope.go:117] "RemoveContainer" containerID="85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7" Sep 10 19:24:46.633297 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:24:46.633267 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7\": container with ID starting with 85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7 not found: ID does not exist" containerID="85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7" Sep 10 19:24:46.633345 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.633305 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7"} err="failed to get container status \"85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7\": rpc error: code = NotFound desc = could not find container \"85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7\": container with ID starting with 85347d66fb83259a46592fe5dcf7a24b3a78e266bb2205d0894bfcd3090637e7 not found: ID does not exist" Sep 10 19:24:46.641145 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.641116 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b59c58d5f-2bz7j"] Sep 10 19:24:46.647517 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:46.647495 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b59c58d5f-2bz7j"] Sep 10 19:24:48.487172 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:24:48.487135 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="160cb959-927f-41be-998d-addcec346bc0" path="/var/lib/kubelet/pods/160cb959-927f-41be-998d-addcec346bc0/volumes" Sep 10 19:25:14.403391 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:25:14.403361 2584 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 19:26:50.565831 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.565792 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-645c954fbc-gscd9"] Sep 10 19:26:50.566360 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.566042 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160cb959-927f-41be-998d-addcec346bc0" containerName="console" Sep 10 19:26:50.566360 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.566054 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="160cb959-927f-41be-998d-addcec346bc0" containerName="console" Sep 10 19:26:50.566360 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.566110 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="160cb959-927f-41be-998d-addcec346bc0" containerName="console" Sep 10 19:26:50.568777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.568759 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.587902 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.587870 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645c954fbc-gscd9"] Sep 10 19:26:50.696470 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.696429 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/111b8e3c-01f1-4797-9164-544f8905a7f8-console-oauth-config\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.696649 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.696476 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-service-ca\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.696649 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.696535 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-trusted-ca-bundle\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.696649 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.696572 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/111b8e3c-01f1-4797-9164-544f8905a7f8-console-serving-cert\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.696649 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.696591 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-oauth-serving-cert\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.696649 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.696608 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-console-config\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.696649 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.696628 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzn4h\" (UniqueName: \"kubernetes.io/projected/111b8e3c-01f1-4797-9164-544f8905a7f8-kube-api-access-xzn4h\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.797258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.797211 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/111b8e3c-01f1-4797-9164-544f8905a7f8-console-serving-cert\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.797258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.797262 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-oauth-serving-cert\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.797398 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.797303 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-console-config\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.797398 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.797326 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzn4h\" (UniqueName: \"kubernetes.io/projected/111b8e3c-01f1-4797-9164-544f8905a7f8-kube-api-access-xzn4h\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.797398 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.797363 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/111b8e3c-01f1-4797-9164-544f8905a7f8-console-oauth-config\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.797504 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.797399 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-service-ca\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.797598 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.797570 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-trusted-ca-bundle\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.798103 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.798080 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-oauth-serving-cert\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.798246 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.798224 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-service-ca\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.798341 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.798224 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-console-config\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.798694 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.798669 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/111b8e3c-01f1-4797-9164-544f8905a7f8-trusted-ca-bundle\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.799936 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.799914 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/111b8e3c-01f1-4797-9164-544f8905a7f8-console-serving-cert\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.800018 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.799950 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/111b8e3c-01f1-4797-9164-544f8905a7f8-console-oauth-config\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.805651 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.805630 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzn4h\" (UniqueName: \"kubernetes.io/projected/111b8e3c-01f1-4797-9164-544f8905a7f8-kube-api-access-xzn4h\") pod \"console-645c954fbc-gscd9\" (UID: \"111b8e3c-01f1-4797-9164-544f8905a7f8\") " pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:50.877615 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:50.877524 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:26:51.155403 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:51.155375 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645c954fbc-gscd9"] Sep 10 19:26:51.158004 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:26:51.157977 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111b8e3c_01f1_4797_9164_544f8905a7f8.slice/crio-0d5d3b06137e49d54e26bfd9bac0c98e2027b97798d92d0bd056dea64c91a18a WatchSource:0}: Error finding container 0d5d3b06137e49d54e26bfd9bac0c98e2027b97798d92d0bd056dea64c91a18a: Status 404 returned error can't find the container with id 0d5d3b06137e49d54e26bfd9bac0c98e2027b97798d92d0bd056dea64c91a18a Sep 10 19:26:52.047886 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:52.047844 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645c954fbc-gscd9" event={"ID":"111b8e3c-01f1-4797-9164-544f8905a7f8","Type":"ContainerStarted","Data":"228ab790786854e3c3f13542e7c911639b3d4a66d28dc04f4dd76369151c38ed"} Sep 10 19:26:52.048303 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:52.047904 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645c954fbc-gscd9" event={"ID":"111b8e3c-01f1-4797-9164-544f8905a7f8","Type":"ContainerStarted","Data":"0d5d3b06137e49d54e26bfd9bac0c98e2027b97798d92d0bd056dea64c91a18a"} Sep 10 19:26:52.066066 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:52.066010 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-645c954fbc-gscd9" podStartSLOduration=2.065991208 podStartE2EDuration="2.065991208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:26:52.064939492 +0000 UTC m=+398.032751739" watchObservedRunningTime="2026-09-10 19:26:52.065991208 +0000 UTC m=+398.033803456" Sep 10 19:26:56.354108 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.354067 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jnnc9"] Sep 10 19:26:56.357246 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.357222 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jnnc9" Sep 10 19:26:56.360205 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.360179 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fdjvv\"" Sep 10 19:26:56.360419 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.360207 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 19:26:56.361050 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.361036 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 19:26:56.375581 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.375550 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jnnc9"] Sep 10 19:26:56.444368 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.444320 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb641ed-7239-4777-9f9a-c15d6cf81e73-bound-sa-token\") pod \"cert-manager-55b4d954d5-jnnc9\" (UID: \"1eb641ed-7239-4777-9f9a-c15d6cf81e73\") " pod="cert-manager/cert-manager-55b4d954d5-jnnc9" Sep 10 19:26:56.444368 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.444377 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qc5r\" (UniqueName: \"kubernetes.io/projected/1eb641ed-7239-4777-9f9a-c15d6cf81e73-kube-api-access-5qc5r\") pod \"cert-manager-55b4d954d5-jnnc9\" (UID: \"1eb641ed-7239-4777-9f9a-c15d6cf81e73\") " pod="cert-manager/cert-manager-55b4d954d5-jnnc9" Sep 10 19:26:56.545458 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.545420 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qc5r\" (UniqueName: \"kubernetes.io/projected/1eb641ed-7239-4777-9f9a-c15d6cf81e73-kube-api-access-5qc5r\") pod \"cert-manager-55b4d954d5-jnnc9\" (UID: \"1eb641ed-7239-4777-9f9a-c15d6cf81e73\") " pod="cert-manager/cert-manager-55b4d954d5-jnnc9" Sep 10 19:26:56.545611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.545506 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb641ed-7239-4777-9f9a-c15d6cf81e73-bound-sa-token\") pod \"cert-manager-55b4d954d5-jnnc9\" (UID: \"1eb641ed-7239-4777-9f9a-c15d6cf81e73\") " pod="cert-manager/cert-manager-55b4d954d5-jnnc9" Sep 10 19:26:56.554185 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.554138 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb641ed-7239-4777-9f9a-c15d6cf81e73-bound-sa-token\") pod \"cert-manager-55b4d954d5-jnnc9\" (UID: \"1eb641ed-7239-4777-9f9a-c15d6cf81e73\") " pod="cert-manager/cert-manager-55b4d954d5-jnnc9" Sep 10 19:26:56.554349 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.554269 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qc5r\" (UniqueName: \"kubernetes.io/projected/1eb641ed-7239-4777-9f9a-c15d6cf81e73-kube-api-access-5qc5r\") pod \"cert-manager-55b4d954d5-jnnc9\" (UID: \"1eb641ed-7239-4777-9f9a-c15d6cf81e73\") " pod="cert-manager/cert-manager-55b4d954d5-jnnc9" Sep 10 19:26:56.666082 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.665992 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jnnc9" Sep 10 19:26:56.940809 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.940729 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jnnc9"] Sep 10 19:26:56.944255 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:26:56.944225 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1eb641ed_7239_4777_9f9a_c15d6cf81e73.slice/crio-f461780c04af1ae46e566d06a83b8ca528e93dc6eba5c86c02a848016d53ef76 WatchSource:0}: Error finding container f461780c04af1ae46e566d06a83b8ca528e93dc6eba5c86c02a848016d53ef76: Status 404 returned error can't find the container with id f461780c04af1ae46e566d06a83b8ca528e93dc6eba5c86c02a848016d53ef76 Sep 10 19:26:56.946598 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:56.946581 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:26:57.067985 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:57.067947 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jnnc9" event={"ID":"1eb641ed-7239-4777-9f9a-c15d6cf81e73","Type":"ContainerStarted","Data":"f461780c04af1ae46e566d06a83b8ca528e93dc6eba5c86c02a848016d53ef76"} Sep 10 19:26:58.775693 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.775659 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz"] Sep 10 19:26:58.780493 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.780454 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" Sep 10 19:26:58.782976 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.782948 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-l9qfs\"" Sep 10 19:26:58.783183 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.783163 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:26:58.783845 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.783823 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 19:26:58.794130 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.794103 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz"] Sep 10 19:26:58.864106 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.864068 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfx7z\" (UniqueName: \"kubernetes.io/projected/5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13-kube-api-access-sfx7z\") pod \"openshift-lws-operator-fd8ccff4c-s4qrz\" (UID: \"5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" Sep 10 19:26:58.864293 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.864134 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s4qrz\" (UID: \"5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" Sep 10 19:26:58.965501 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.965465 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfx7z\" (UniqueName: \"kubernetes.io/projected/5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13-kube-api-access-sfx7z\") pod \"openshift-lws-operator-fd8ccff4c-s4qrz\" (UID: \"5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" Sep 10 19:26:58.965678 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.965516 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s4qrz\" (UID: \"5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" Sep 10 19:26:58.965902 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.965882 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s4qrz\" (UID: \"5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" Sep 10 19:26:58.977090 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:58.977061 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfx7z\" (UniqueName: \"kubernetes.io/projected/5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13-kube-api-access-sfx7z\") pod \"openshift-lws-operator-fd8ccff4c-s4qrz\" (UID: \"5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" Sep 10 19:26:59.091717 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:26:59.091671 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" Sep 10 19:27:00.080745 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:00.080706 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jnnc9" event={"ID":"1eb641ed-7239-4777-9f9a-c15d6cf81e73","Type":"ContainerStarted","Data":"c60e8ac0678e49c1f595234645cc6ff1bdc36799902342a8bb6a6a3d9a598a24"} Sep 10 19:27:00.101706 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:00.101595 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jnnc9" podStartSLOduration=1.133901287 podStartE2EDuration="4.101577904s" totalImagesPullingTime="2.967676617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:00.100186347 +0000 UTC m=+406.067998593" watchObservedRunningTime="2026-09-10 19:27:00.101577904 +0000 UTC m=+406.069390150" Sep 10 19:27:00.120987 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:00.120955 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz"] Sep 10 19:27:00.124037 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:27:00.124008 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f6a3d7f_f703_49d5_a1d9_b8d6c0ceac13.slice/crio-02e4b8eba05c26cfd868e3688cf4dd4904fd82be30d8091c87768a73e7b9e425 WatchSource:0}: Error finding container 02e4b8eba05c26cfd868e3688cf4dd4904fd82be30d8091c87768a73e7b9e425: Status 404 returned error can't find the container with id 02e4b8eba05c26cfd868e3688cf4dd4904fd82be30d8091c87768a73e7b9e425 Sep 10 19:27:00.878333 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:00.878273 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:27:00.878697 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:00.878640 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:27:00.884625 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:00.884595 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:27:01.087155 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:01.087107 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" event={"ID":"5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13","Type":"ContainerStarted","Data":"02e4b8eba05c26cfd868e3688cf4dd4904fd82be30d8091c87768a73e7b9e425"} Sep 10 19:27:01.092667 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:01.092632 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-645c954fbc-gscd9" Sep 10 19:27:01.147687 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:01.147564 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c8ff65d6d-m2hmx"] Sep 10 19:27:03.096727 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:03.096684 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" event={"ID":"5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13","Type":"ContainerStarted","Data":"d72b979bdaca76af8fa546de31db873a4da27185cc26c4eafc15f0d2bc35f8c2"} Sep 10 19:27:03.128539 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:03.128469 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4qrz" podStartSLOduration=2.686439203 podStartE2EDuration="5.128446436s" totalImagesPullingTime="2.442007233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:03.127641905 +0000 UTC m=+409.095454155" watchObservedRunningTime="2026-09-10 19:27:03.128446436 +0000 UTC m=+409.096258679" Sep 10 19:27:20.248059 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.248018 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb"] Sep 10 19:27:20.251315 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.251266 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.259529 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.259491 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hws6j\"" Sep 10 19:27:20.259660 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.259558 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 19:27:20.259660 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.259571 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 19:27:20.259660 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.259568 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 19:27:20.274586 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.274550 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb"] Sep 10 19:27:20.340649 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.340608 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv92h\" (UniqueName: \"kubernetes.io/projected/a491f76f-cfef-499a-9b1c-b11615af46f0-kube-api-access-hv92h\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.340841 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.340657 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a491f76f-cfef-499a-9b1c-b11615af46f0-manager-config\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.340841 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.340677 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a491f76f-cfef-499a-9b1c-b11615af46f0-cert\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.340841 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.340802 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a491f76f-cfef-499a-9b1c-b11615af46f0-metrics-cert\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.441394 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.441349 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a491f76f-cfef-499a-9b1c-b11615af46f0-manager-config\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.441394 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.441394 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a491f76f-cfef-499a-9b1c-b11615af46f0-cert\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.441646 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.441461 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a491f76f-cfef-499a-9b1c-b11615af46f0-metrics-cert\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.441646 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.441504 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv92h\" (UniqueName: \"kubernetes.io/projected/a491f76f-cfef-499a-9b1c-b11615af46f0-kube-api-access-hv92h\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.442189 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.442157 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a491f76f-cfef-499a-9b1c-b11615af46f0-manager-config\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.443958 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.443938 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a491f76f-cfef-499a-9b1c-b11615af46f0-cert\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.444054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.443987 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a491f76f-cfef-499a-9b1c-b11615af46f0-metrics-cert\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.480366 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.480327 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv92h\" (UniqueName: \"kubernetes.io/projected/a491f76f-cfef-499a-9b1c-b11615af46f0-kube-api-access-hv92h\") pod \"lws-controller-manager-57fccbbc8-k9nvb\" (UID: \"a491f76f-cfef-499a-9b1c-b11615af46f0\") " pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.560501 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.560471 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:20.851120 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:20.851091 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb"] Sep 10 19:27:20.853421 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:27:20.853390 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda491f76f_cfef_499a_9b1c_b11615af46f0.slice/crio-75f446631d3b83e0dfef0380ea8a52a18410077df23f3bf62498d51de5605c38 WatchSource:0}: Error finding container 75f446631d3b83e0dfef0380ea8a52a18410077df23f3bf62498d51de5605c38: Status 404 returned error can't find the container with id 75f446631d3b83e0dfef0380ea8a52a18410077df23f3bf62498d51de5605c38 Sep 10 19:27:21.165181 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:21.165093 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" event={"ID":"a491f76f-cfef-499a-9b1c-b11615af46f0","Type":"ContainerStarted","Data":"75f446631d3b83e0dfef0380ea8a52a18410077df23f3bf62498d51de5605c38"} Sep 10 19:27:22.576758 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.576726 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz"] Sep 10 19:27:22.580399 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.580374 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.587465 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.587147 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 19:27:22.588017 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.587816 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 19:27:22.588017 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.588014 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 19:27:22.589726 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.589708 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q6dtv\"" Sep 10 19:27:22.595442 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.595421 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 19:27:22.622898 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.622860 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz"] Sep 10 19:27:22.760245 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.760201 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19a7ac54-3c02-42ac-959a-a37752df52f7-manifests\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.760245 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.760248 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhw85\" (UniqueName: \"kubernetes.io/projected/19a7ac54-3c02-42ac-959a-a37752df52f7-kube-api-access-lhw85\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.760506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.760273 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19a7ac54-3c02-42ac-959a-a37752df52f7-tmp\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.760506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.760377 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19a7ac54-3c02-42ac-959a-a37752df52f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.760506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.760418 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19a7ac54-3c02-42ac-959a-a37752df52f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.861223 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.861136 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19a7ac54-3c02-42ac-959a-a37752df52f7-tmp\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.861223 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.861195 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19a7ac54-3c02-42ac-959a-a37752df52f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.861223 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.861219 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19a7ac54-3c02-42ac-959a-a37752df52f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.861485 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.861249 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19a7ac54-3c02-42ac-959a-a37752df52f7-manifests\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.861485 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.861271 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhw85\" (UniqueName: \"kubernetes.io/projected/19a7ac54-3c02-42ac-959a-a37752df52f7-kube-api-access-lhw85\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.861753 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.861728 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19a7ac54-3c02-42ac-959a-a37752df52f7-manifests\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.863469 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.863442 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19a7ac54-3c02-42ac-959a-a37752df52f7-tmp\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.863695 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.863677 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19a7ac54-3c02-42ac-959a-a37752df52f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.863737 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.863703 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19a7ac54-3c02-42ac-959a-a37752df52f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.873201 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.873173 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhw85\" (UniqueName: \"kubernetes.io/projected/19a7ac54-3c02-42ac-959a-a37752df52f7-kube-api-access-lhw85\") pod \"opendatahub-operator-controller-manager-5ddcddd787-wmrjz\" (UID: \"19a7ac54-3c02-42ac-959a-a37752df52f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:22.891034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:22.890997 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:23.174856 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:23.174765 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" event={"ID":"a491f76f-cfef-499a-9b1c-b11615af46f0","Type":"ContainerStarted","Data":"c71305eca5faeaffabc87371cb94ec8f3ccfc44e1520b8a57e832f3298b60275"} Sep 10 19:27:23.175008 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:23.174916 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:23.179420 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:23.179263 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz"] Sep 10 19:27:23.182157 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:27:23.182131 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a7ac54_3c02_42ac_959a_a37752df52f7.slice/crio-6f81bd04cb3a2201c7a94c32f88f2ecb12e478bdb2af9ceabc6456b9743e5cf7 WatchSource:0}: Error finding container 6f81bd04cb3a2201c7a94c32f88f2ecb12e478bdb2af9ceabc6456b9743e5cf7: Status 404 returned error can't find the container with id 6f81bd04cb3a2201c7a94c32f88f2ecb12e478bdb2af9ceabc6456b9743e5cf7 Sep 10 19:27:23.200825 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:23.200768 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" podStartSLOduration=1.773460796 podStartE2EDuration="3.200749435s" totalImagesPullingTime="1.427288639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:23.200122364 +0000 UTC m=+429.167934611" watchObservedRunningTime="2026-09-10 19:27:23.200749435 +0000 UTC m=+429.168561793" Sep 10 19:27:24.182117 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:24.182058 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" event={"ID":"19a7ac54-3c02-42ac-959a-a37752df52f7","Type":"ContainerStarted","Data":"6f81bd04cb3a2201c7a94c32f88f2ecb12e478bdb2af9ceabc6456b9743e5cf7"} Sep 10 19:27:26.174407 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.174372 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c8ff65d6d-m2hmx" podUID="bd77b55c-e8fe-4654-b2c0-59abae29ae6f" containerName="console" containerID="cri-o://51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db" gracePeriod=15 Sep 10 19:27:26.430823 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.430795 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:27:26.493691 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.493662 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-config\") pod \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " Sep 10 19:27:26.493850 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.493705 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-oauth-serving-cert\") pod \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " Sep 10 19:27:26.493850 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.493725 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlxvb\" (UniqueName: \"kubernetes.io/projected/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-kube-api-access-vlxvb\") pod \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " Sep 10 19:27:26.494102 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.494073 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-config" (OuterVolumeSpecName: "console-config") pod "bd77b55c-e8fe-4654-b2c0-59abae29ae6f" (UID: "bd77b55c-e8fe-4654-b2c0-59abae29ae6f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:27:26.494189 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.494176 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bd77b55c-e8fe-4654-b2c0-59abae29ae6f" (UID: "bd77b55c-e8fe-4654-b2c0-59abae29ae6f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:27:26.495895 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.495875 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-kube-api-access-vlxvb" (OuterVolumeSpecName: "kube-api-access-vlxvb") pod "bd77b55c-e8fe-4654-b2c0-59abae29ae6f" (UID: "bd77b55c-e8fe-4654-b2c0-59abae29ae6f"). InnerVolumeSpecName "kube-api-access-vlxvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:27:26.594755 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.594709 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-trusted-ca-bundle\") pod \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " Sep 10 19:27:26.594956 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.594791 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-service-ca\") pod \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " Sep 10 19:27:26.594956 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.594854 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-oauth-config\") pod \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " Sep 10 19:27:26.594956 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.594884 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-serving-cert\") pod \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\" (UID: \"bd77b55c-e8fe-4654-b2c0-59abae29ae6f\") " Sep 10 19:27:26.595111 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.595078 2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-config\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:27:26.595111 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.595095 2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-oauth-serving-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:27:26.595184 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.595109 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlxvb\" (UniqueName: \"kubernetes.io/projected/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-kube-api-access-vlxvb\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:27:26.595184 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.595153 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bd77b55c-e8fe-4654-b2c0-59abae29ae6f" (UID: "bd77b55c-e8fe-4654-b2c0-59abae29ae6f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:27:26.595251 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.595200 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-service-ca" (OuterVolumeSpecName: "service-ca") pod "bd77b55c-e8fe-4654-b2c0-59abae29ae6f" (UID: "bd77b55c-e8fe-4654-b2c0-59abae29ae6f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:27:26.597007 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.596986 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bd77b55c-e8fe-4654-b2c0-59abae29ae6f" (UID: "bd77b55c-e8fe-4654-b2c0-59abae29ae6f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:27:26.597074 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.597010 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bd77b55c-e8fe-4654-b2c0-59abae29ae6f" (UID: "bd77b55c-e8fe-4654-b2c0-59abae29ae6f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:27:26.695666 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.695568 2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-trusted-ca-bundle\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:27:26.695666 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.695603 2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-service-ca\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:27:26.695666 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.695616 2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-oauth-config\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:27:26.695666 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:26.695628 2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd77b55c-e8fe-4654-b2c0-59abae29ae6f-console-serving-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:27:27.195997 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:27.195953 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" event={"ID":"19a7ac54-3c02-42ac-959a-a37752df52f7","Type":"ContainerStarted","Data":"7bf9bb2382c855c284e58a79a7fed533abfcd0792714ccd019b64a46d073b16c"} Sep 10 19:27:27.197337 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:27.197306 2584 generic.go:356] "Generic (PLEG): container finished" podID="bd77b55c-e8fe-4654-b2c0-59abae29ae6f" containerID="51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db" exitCode=0 Sep 10 19:27:27.197436 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:27.197350 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8ff65d6d-m2hmx" event={"ID":"bd77b55c-e8fe-4654-b2c0-59abae29ae6f","Type":"ContainerDied","Data":"51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db"} Sep 10 19:27:27.197436 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:27.197380 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8ff65d6d-m2hmx" event={"ID":"bd77b55c-e8fe-4654-b2c0-59abae29ae6f","Type":"ContainerDied","Data":"717de21fadbe97f17c03f347fafbb04888bbaea5fc3e11a7da4e1c9e5fe58114"} Sep 10 19:27:27.197436 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:27.197402 2584 scope.go:117] "RemoveContainer" containerID="51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db" Sep 10 19:27:27.197436 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:27.197422 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8ff65d6d-m2hmx" Sep 10 19:27:27.206151 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:27.206131 2584 scope.go:117] "RemoveContainer" containerID="51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db" Sep 10 19:27:27.206478 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:27:27.206456 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db\": container with ID starting with 51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db not found: ID does not exist" containerID="51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db" Sep 10 19:27:27.206558 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:27.206483 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db"} err="failed to get container status \"51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db\": rpc error: code = NotFound desc = could not find container \"51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db\": container with ID starting with 51f61892b7bc4ddc30a5c67dc934cf55bdbd8d021126613a544f6f2da73ce3db not found: ID does not exist" Sep 10 19:27:27.237840 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:27.237798 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c8ff65d6d-m2hmx"] Sep 10 19:27:27.244190 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:27.244160 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c8ff65d6d-m2hmx"] Sep 10 19:27:28.205993 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:28.205954 2584 generic.go:356] "Generic (PLEG): container finished" podID="19a7ac54-3c02-42ac-959a-a37752df52f7" containerID="7bf9bb2382c855c284e58a79a7fed533abfcd0792714ccd019b64a46d073b16c" exitCode=0 Sep 10 19:27:28.206493 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:28.206041 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" event={"ID":"19a7ac54-3c02-42ac-959a-a37752df52f7","Type":"ContainerDied","Data":"7bf9bb2382c855c284e58a79a7fed533abfcd0792714ccd019b64a46d073b16c"} Sep 10 19:27:28.487613 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:28.487533 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd77b55c-e8fe-4654-b2c0-59abae29ae6f" path="/var/lib/kubelet/pods/bd77b55c-e8fe-4654-b2c0-59abae29ae6f/volumes" Sep 10 19:27:29.214109 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:29.214009 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" event={"ID":"19a7ac54-3c02-42ac-959a-a37752df52f7","Type":"ContainerStarted","Data":"ab35479564e9f72c32ecf0230c5b9aa00d33961535ba7817c5c3a8c020e4be2e"} Sep 10 19:27:29.214538 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:29.214146 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:29.243579 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:29.243523 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" podStartSLOduration=2.484196701 podStartE2EDuration="7.243505335s" totalImagesPullingTime="3.629519882s" totalInitContainerRuntime="1.129788752s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:29.241822338 +0000 UTC m=+435.209634598" watchObservedRunningTime="2026-09-10 19:27:29.243505335 +0000 UTC m=+435.211317581" Sep 10 19:27:34.184913 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:34.184881 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57fccbbc8-k9nvb" Sep 10 19:27:40.221399 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:40.221320 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5ddcddd787-wmrjz" Sep 10 19:27:50.742128 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.742091 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj"] Sep 10 19:27:50.742624 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.742499 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd77b55c-e8fe-4654-b2c0-59abae29ae6f" containerName="console" Sep 10 19:27:50.742624 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.742515 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd77b55c-e8fe-4654-b2c0-59abae29ae6f" containerName="console" Sep 10 19:27:50.742624 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.742598 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd77b55c-e8fe-4654-b2c0-59abae29ae6f" containerName="console" Sep 10 19:27:50.744780 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.744747 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.747696 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.747668 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 19:27:50.747829 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.747717 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5djps\"" Sep 10 19:27:50.748029 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.748011 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 19:27:50.748121 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.748016 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 19:27:50.758434 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.758403 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj"] Sep 10 19:27:50.791686 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.791647 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.791686 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.791702 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.791959 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.791777 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.791959 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.791824 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.791959 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.791853 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8fm7\" (UniqueName: \"kubernetes.io/projected/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-kube-api-access-c8fm7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.791959 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.791913 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.791959 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.791940 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.792200 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.792003 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.792200 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.792024 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893247 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893211 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893247 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893252 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8fm7\" (UniqueName: \"kubernetes.io/projected/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-kube-api-access-c8fm7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893292 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893311 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893337 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893401 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893462 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893527 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893824 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893584 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893824 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893710 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893824 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893750 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893968 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893894 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.893968 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.893930 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.894226 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.894205 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.895811 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.895787 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.895892 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.895814 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.908724 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.908690 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:50.908890 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:50.908869 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8fm7\" (UniqueName: \"kubernetes.io/projected/4f37593c-597c-41ac-ae4c-4f4e3a819ba5-kube-api-access-c8fm7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj\" (UID: \"4f37593c-597c-41ac-ae4c-4f4e3a819ba5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:51.055807 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:51.055767 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:27:51.332104 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:51.331937 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj"] Sep 10 19:27:51.334648 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:27:51.334622 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f37593c_597c_41ac_ae4c_4f4e3a819ba5.slice/crio-bab851741dbd7eb752ce5d53ca418e7baeb25e15cd5e87206d490ee957f6847f WatchSource:0}: Error finding container bab851741dbd7eb752ce5d53ca418e7baeb25e15cd5e87206d490ee957f6847f: Status 404 returned error can't find the container with id bab851741dbd7eb752ce5d53ca418e7baeb25e15cd5e87206d490ee957f6847f Sep 10 19:27:52.310824 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:52.310790 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" event={"ID":"4f37593c-597c-41ac-ae4c-4f4e3a819ba5","Type":"ContainerStarted","Data":"bab851741dbd7eb752ce5d53ca418e7baeb25e15cd5e87206d490ee957f6847f"} Sep 10 19:27:53.930871 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:53.930839 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-md5v2"] Sep 10 19:27:53.937341 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:53.935509 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" Sep 10 19:27:53.939069 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:53.939045 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 19:27:53.962911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:53.962882 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-md5v2"] Sep 10 19:27:54.022631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:54.022583 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tmw2\" (UniqueName: \"kubernetes.io/projected/fbd116bd-2928-48ef-9990-b3766580e6c7-kube-api-access-8tmw2\") pod \"model-serving-api-54d89bd5fc-md5v2\" (UID: \"fbd116bd-2928-48ef-9990-b3766580e6c7\") " pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" Sep 10 19:27:54.022816 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:54.022645 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd116bd-2928-48ef-9990-b3766580e6c7-tls-certs\") pod \"model-serving-api-54d89bd5fc-md5v2\" (UID: \"fbd116bd-2928-48ef-9990-b3766580e6c7\") " pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" Sep 10 19:27:54.123872 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:54.123831 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tmw2\" (UniqueName: \"kubernetes.io/projected/fbd116bd-2928-48ef-9990-b3766580e6c7-kube-api-access-8tmw2\") pod \"model-serving-api-54d89bd5fc-md5v2\" (UID: \"fbd116bd-2928-48ef-9990-b3766580e6c7\") " pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" Sep 10 19:27:54.124070 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:54.123883 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd116bd-2928-48ef-9990-b3766580e6c7-tls-certs\") pod \"model-serving-api-54d89bd5fc-md5v2\" (UID: \"fbd116bd-2928-48ef-9990-b3766580e6c7\") " pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" Sep 10 19:27:54.126324 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:54.126300 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd116bd-2928-48ef-9990-b3766580e6c7-tls-certs\") pod \"model-serving-api-54d89bd5fc-md5v2\" (UID: \"fbd116bd-2928-48ef-9990-b3766580e6c7\") " pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" Sep 10 19:27:54.140141 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:54.140107 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tmw2\" (UniqueName: \"kubernetes.io/projected/fbd116bd-2928-48ef-9990-b3766580e6c7-kube-api-access-8tmw2\") pod \"model-serving-api-54d89bd5fc-md5v2\" (UID: \"fbd116bd-2928-48ef-9990-b3766580e6c7\") " pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" Sep 10 19:27:54.247672 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:54.247570 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" Sep 10 19:27:54.524452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:54.524371 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-md5v2"] Sep 10 19:27:54.527256 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:27:54.527227 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd116bd_2928_48ef_9990_b3766580e6c7.slice/crio-cd5a86bb81ce017fa2ec161ed7316cf5df4061c013e35502bd308000a0f2f0a3 WatchSource:0}: Error finding container cd5a86bb81ce017fa2ec161ed7316cf5df4061c013e35502bd308000a0f2f0a3: Status 404 returned error can't find the container with id cd5a86bb81ce017fa2ec161ed7316cf5df4061c013e35502bd308000a0f2f0a3 Sep 10 19:27:55.328035 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:55.327989 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" event={"ID":"fbd116bd-2928-48ef-9990-b3766580e6c7","Type":"ContainerStarted","Data":"cd5a86bb81ce017fa2ec161ed7316cf5df4061c013e35502bd308000a0f2f0a3"} Sep 10 19:27:57.338312 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:57.338255 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" event={"ID":"fbd116bd-2928-48ef-9990-b3766580e6c7","Type":"ContainerStarted","Data":"135ccc2da8232095935b02ebd4f24828f924f71bf06d8f936e76ff86f092d2f3"} Sep 10 19:27:57.338673 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:57.338344 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" Sep 10 19:27:57.357518 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:27:57.357462 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" podStartSLOduration=1.6627823400000001 podStartE2EDuration="4.357439327s" totalImagesPullingTime="2.694656987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:57.355875617 +0000 UTC m=+463.323687875" watchObservedRunningTime="2026-09-10 19:27:57.357439327 +0000 UTC m=+463.325251574" Sep 10 19:28:01.632865 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:01.632822 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:28:01.633144 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:01.632901 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:28:01.633144 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:01.632928 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:28:02.366913 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:02.366874 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" event={"ID":"4f37593c-597c-41ac-ae4c-4f4e3a819ba5","Type":"ContainerStarted","Data":"2c0db875fa854cff1d597216c4d84e3564154eb4caaa57b807260ad3dc62dc32"} Sep 10 19:28:02.406504 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:02.406449 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" podStartSLOduration=2.110516048 podStartE2EDuration="12.406432068s" totalImagesPullingTime="10.29591602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:02.403681943 +0000 UTC m=+468.371494190" watchObservedRunningTime="2026-09-10 19:28:02.406432068 +0000 UTC m=+468.374244339" Sep 10 19:28:03.056986 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:03.056944 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:28:03.061837 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:03.061814 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:28:03.371689 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:03.371591 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:28:03.372542 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:03.372521 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj" Sep 10 19:28:05.954645 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:05.954611 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8"] Sep 10 19:28:05.957028 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:05.957011 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" Sep 10 19:28:05.963950 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:05.963918 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 19:28:05.964203 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:05.964188 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-hgjpn\"" Sep 10 19:28:05.993167 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:05.993127 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f"] Sep 10 19:28:05.995526 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:05.995509 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:28:05.998272 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:05.998248 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 19:28:05.999096 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:05.999080 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hvc6b\"" Sep 10 19:28:06.088292 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.088230 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8"] Sep 10 19:28:06.124117 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.124071 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07c67934-a96f-45de-b287-a4933090904d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q556f\" (UID: \"07c67934-a96f-45de-b287-a4933090904d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:28:06.124117 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.124117 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d18976d-a5b7-484c-b36d-6bde0096be99-cert\") pod \"kserve-controller-manager-d6f4b65b8-xx9l8\" (UID: \"3d18976d-a5b7-484c-b36d-6bde0096be99\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" Sep 10 19:28:06.124377 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.124149 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z879\" (UniqueName: \"kubernetes.io/projected/07c67934-a96f-45de-b287-a4933090904d-kube-api-access-5z879\") pod \"llmisvc-controller-manager-777c6bd7d5-q556f\" (UID: \"07c67934-a96f-45de-b287-a4933090904d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:28:06.124377 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.124240 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsjx7\" (UniqueName: \"kubernetes.io/projected/3d18976d-a5b7-484c-b36d-6bde0096be99-kube-api-access-nsjx7\") pod \"kserve-controller-manager-d6f4b65b8-xx9l8\" (UID: \"3d18976d-a5b7-484c-b36d-6bde0096be99\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" Sep 10 19:28:06.155604 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.155565 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f"] Sep 10 19:28:06.225498 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.225403 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07c67934-a96f-45de-b287-a4933090904d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q556f\" (UID: \"07c67934-a96f-45de-b287-a4933090904d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:28:06.225498 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.225445 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d18976d-a5b7-484c-b36d-6bde0096be99-cert\") pod \"kserve-controller-manager-d6f4b65b8-xx9l8\" (UID: \"3d18976d-a5b7-484c-b36d-6bde0096be99\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" Sep 10 19:28:06.225498 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.225468 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z879\" (UniqueName: \"kubernetes.io/projected/07c67934-a96f-45de-b287-a4933090904d-kube-api-access-5z879\") pod \"llmisvc-controller-manager-777c6bd7d5-q556f\" (UID: \"07c67934-a96f-45de-b287-a4933090904d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:28:06.225498 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.225497 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsjx7\" (UniqueName: \"kubernetes.io/projected/3d18976d-a5b7-484c-b36d-6bde0096be99-kube-api-access-nsjx7\") pod \"kserve-controller-manager-d6f4b65b8-xx9l8\" (UID: \"3d18976d-a5b7-484c-b36d-6bde0096be99\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" Sep 10 19:28:06.225777 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:28:06.225567 2584 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 10 19:28:06.225777 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:28:06.225656 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07c67934-a96f-45de-b287-a4933090904d-cert podName:07c67934-a96f-45de-b287-a4933090904d nodeName:}" failed. No retries permitted until 2026-09-10 19:28:06.725630437 +0000 UTC m=+472.693442662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/07c67934-a96f-45de-b287-a4933090904d-cert") pod "llmisvc-controller-manager-777c6bd7d5-q556f" (UID: "07c67934-a96f-45de-b287-a4933090904d") : secret "llmisvc-webhook-server-cert" not found Sep 10 19:28:06.228045 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.228020 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d18976d-a5b7-484c-b36d-6bde0096be99-cert\") pod \"kserve-controller-manager-d6f4b65b8-xx9l8\" (UID: \"3d18976d-a5b7-484c-b36d-6bde0096be99\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" Sep 10 19:28:06.239614 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.239580 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z879\" (UniqueName: \"kubernetes.io/projected/07c67934-a96f-45de-b287-a4933090904d-kube-api-access-5z879\") pod \"llmisvc-controller-manager-777c6bd7d5-q556f\" (UID: \"07c67934-a96f-45de-b287-a4933090904d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:28:06.241067 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.241042 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsjx7\" (UniqueName: \"kubernetes.io/projected/3d18976d-a5b7-484c-b36d-6bde0096be99-kube-api-access-nsjx7\") pod \"kserve-controller-manager-d6f4b65b8-xx9l8\" (UID: \"3d18976d-a5b7-484c-b36d-6bde0096be99\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" Sep 10 19:28:06.266931 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.266897 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" Sep 10 19:28:06.550655 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.550575 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8"] Sep 10 19:28:06.554996 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:28:06.554954 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d18976d_a5b7_484c_b36d_6bde0096be99.slice/crio-214fe8b5c7dc673f9f12e750ce88d723c560783b20aa28ee2dba5b928276aa83 WatchSource:0}: Error finding container 214fe8b5c7dc673f9f12e750ce88d723c560783b20aa28ee2dba5b928276aa83: Status 404 returned error can't find the container with id 214fe8b5c7dc673f9f12e750ce88d723c560783b20aa28ee2dba5b928276aa83 Sep 10 19:28:06.730802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.730756 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07c67934-a96f-45de-b287-a4933090904d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q556f\" (UID: \"07c67934-a96f-45de-b287-a4933090904d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:28:06.733299 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.733254 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07c67934-a96f-45de-b287-a4933090904d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q556f\" (UID: \"07c67934-a96f-45de-b287-a4933090904d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:28:06.905905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:06.905865 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:28:07.184945 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:07.184919 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f"] Sep 10 19:28:07.186975 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:28:07.186936 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07c67934_a96f_45de_b287_a4933090904d.slice/crio-1791682c9ef36c6ca0e176d6e750a36f580642a44cca2259f463a08afaa75bd7 WatchSource:0}: Error finding container 1791682c9ef36c6ca0e176d6e750a36f580642a44cca2259f463a08afaa75bd7: Status 404 returned error can't find the container with id 1791682c9ef36c6ca0e176d6e750a36f580642a44cca2259f463a08afaa75bd7 Sep 10 19:28:07.390246 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:07.390199 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" event={"ID":"3d18976d-a5b7-484c-b36d-6bde0096be99","Type":"ContainerStarted","Data":"214fe8b5c7dc673f9f12e750ce88d723c560783b20aa28ee2dba5b928276aa83"} Sep 10 19:28:07.391633 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:07.391593 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" event={"ID":"07c67934-a96f-45de-b287-a4933090904d","Type":"ContainerStarted","Data":"1791682c9ef36c6ca0e176d6e750a36f580642a44cca2259f463a08afaa75bd7"} Sep 10 19:28:08.347940 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:08.347906 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-md5v2" Sep 10 19:28:09.403066 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:09.402969 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" event={"ID":"3d18976d-a5b7-484c-b36d-6bde0096be99","Type":"ContainerStarted","Data":"5e8befa8332c573fe0050314b1e53a6e0b1c532cb612873cf1c8acd836dc9929"} Sep 10 19:28:09.403452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:09.403111 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" Sep 10 19:28:09.448356 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:09.448268 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" podStartSLOduration=2.102487633 podStartE2EDuration="4.448243944s" totalImagesPullingTime="2.345756311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:09.446643221 +0000 UTC m=+475.414455490" watchObservedRunningTime="2026-09-10 19:28:09.448243944 +0000 UTC m=+475.416056192" Sep 10 19:28:11.413601 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:11.413561 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" event={"ID":"07c67934-a96f-45de-b287-a4933090904d","Type":"ContainerStarted","Data":"bdb06d1f71fc7b47cf701baf8211c52479ffa19546d97f2f43642774a09f4fb8"} Sep 10 19:28:11.414017 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:11.413650 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:28:11.462882 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:11.462819 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" podStartSLOduration=3.19930702 podStartE2EDuration="6.462800175s" totalImagesPullingTime="3.263493155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:11.46170238 +0000 UTC m=+477.429514626" watchObservedRunningTime="2026-09-10 19:28:11.462800175 +0000 UTC m=+477.430612440" Sep 10 19:28:40.413046 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:40.413005 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xx9l8" Sep 10 19:28:42.421037 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:28:42.420999 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q556f" Sep 10 19:29:18.493739 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:18.493654 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw"] Sep 10 19:29:18.496926 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:18.496906 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:18.499797 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:18.499774 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lzsjf\"" Sep 10 19:29:18.499994 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:18.499974 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 19:29:18.500713 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:18.500696 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 19:29:18.508927 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:18.508901 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw"] Sep 10 19:29:18.637195 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:18.637151 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-882n2\" (UniqueName: \"kubernetes.io/projected/fec21d8f-2b08-4a72-823c-dfbdd8d8d135-kube-api-access-882n2\") pod \"limitador-operator-controller-manager-74495c69f8-86nlw\" (UID: \"fec21d8f-2b08-4a72-823c-dfbdd8d8d135\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:18.737893 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:18.737849 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882n2\" (UniqueName: \"kubernetes.io/projected/fec21d8f-2b08-4a72-823c-dfbdd8d8d135-kube-api-access-882n2\") pod \"limitador-operator-controller-manager-74495c69f8-86nlw\" (UID: \"fec21d8f-2b08-4a72-823c-dfbdd8d8d135\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:18.746385 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:18.746310 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-882n2\" (UniqueName: \"kubernetes.io/projected/fec21d8f-2b08-4a72-823c-dfbdd8d8d135-kube-api-access-882n2\") pod \"limitador-operator-controller-manager-74495c69f8-86nlw\" (UID: \"fec21d8f-2b08-4a72-823c-dfbdd8d8d135\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:18.807010 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:18.806970 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:19.097645 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:29:19.097613 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec21d8f_2b08_4a72_823c_dfbdd8d8d135.slice/crio-8a533a2551e74960f4a256cf7c63bfa679bf15713b7e7e498daffd26db361df4 WatchSource:0}: Error finding container 8a533a2551e74960f4a256cf7c63bfa679bf15713b7e7e498daffd26db361df4: Status 404 returned error can't find the container with id 8a533a2551e74960f4a256cf7c63bfa679bf15713b7e7e498daffd26db361df4 Sep 10 19:29:19.098141 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:19.098114 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw"] Sep 10 19:29:19.698997 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:19.698957 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" event={"ID":"fec21d8f-2b08-4a72-823c-dfbdd8d8d135","Type":"ContainerStarted","Data":"8a533a2551e74960f4a256cf7c63bfa679bf15713b7e7e498daffd26db361df4"} Sep 10 19:29:21.709510 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:21.709469 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" event={"ID":"fec21d8f-2b08-4a72-823c-dfbdd8d8d135","Type":"ContainerStarted","Data":"acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30"} Sep 10 19:29:21.709890 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:21.709655 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:21.768493 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:21.768439 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" podStartSLOduration=1.251215778 podStartE2EDuration="3.768421968s" totalImagesPullingTime="2.51720619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:21.76645225 +0000 UTC m=+547.734264496" watchObservedRunningTime="2026-09-10 19:29:21.768421968 +0000 UTC m=+547.736234215" Sep 10 19:29:32.434480 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.434436 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw"] Sep 10 19:29:32.435057 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.434683 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" podUID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" containerName="manager" containerID="cri-o://acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30" gracePeriod=2 Sep 10 19:29:32.436805 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.436778 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:32.451516 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.451487 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw"] Sep 10 19:29:32.468051 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:29:32.468005 2584 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fec21d8f-2b08-4a72-823c-dfbdd8d8d135\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-86nlw\": pods \"limitador-operator-controller-manager-74495c69f8-86nlw\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:32.470500 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.470465 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9"] Sep 10 19:29:32.470860 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:29:32.470827 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-86nlw\" is forbidden: User \"system:node:ip-10-0-134-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-64.ec2.internal' and this object" podUID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:32.470961 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.470937 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" containerName="manager" Sep 10 19:29:32.470961 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.470954 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" containerName="manager" Sep 10 19:29:32.471031 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.471014 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" containerName="manager" Sep 10 19:29:32.474270 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.474249 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9" Sep 10 19:29:32.477272 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:29:32.477247 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-86nlw\" is forbidden: User \"system:node:ip-10-0-134-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-64.ec2.internal' and this object" podUID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:32.510609 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.508947 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9"] Sep 10 19:29:32.535637 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.535596 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttp88\" (UniqueName: \"kubernetes.io/projected/b81f8072-fd40-4342-85cb-afe7956ddb62-kube-api-access-ttp88\") pod \"limitador-operator-controller-manager-74495c69f8-zkph9\" (UID: \"b81f8072-fd40-4342-85cb-afe7956ddb62\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9" Sep 10 19:29:32.636860 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.636808 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttp88\" (UniqueName: \"kubernetes.io/projected/b81f8072-fd40-4342-85cb-afe7956ddb62-kube-api-access-ttp88\") pod \"limitador-operator-controller-manager-74495c69f8-zkph9\" (UID: \"b81f8072-fd40-4342-85cb-afe7956ddb62\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9" Sep 10 19:29:32.646210 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.646179 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttp88\" (UniqueName: \"kubernetes.io/projected/b81f8072-fd40-4342-85cb-afe7956ddb62-kube-api-access-ttp88\") pod \"limitador-operator-controller-manager-74495c69f8-zkph9\" (UID: \"b81f8072-fd40-4342-85cb-afe7956ddb62\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9" Sep 10 19:29:32.667018 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.666990 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:32.737258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.737168 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-882n2\" (UniqueName: \"kubernetes.io/projected/fec21d8f-2b08-4a72-823c-dfbdd8d8d135-kube-api-access-882n2\") pod \"fec21d8f-2b08-4a72-823c-dfbdd8d8d135\" (UID: \"fec21d8f-2b08-4a72-823c-dfbdd8d8d135\") " Sep 10 19:29:32.739301 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.739244 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec21d8f-2b08-4a72-823c-dfbdd8d8d135-kube-api-access-882n2" (OuterVolumeSpecName: "kube-api-access-882n2") pod "fec21d8f-2b08-4a72-823c-dfbdd8d8d135" (UID: "fec21d8f-2b08-4a72-823c-dfbdd8d8d135"). InnerVolumeSpecName "kube-api-access-882n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:29:32.759336 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.759299 2584 generic.go:356] "Generic (PLEG): container finished" podID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" containerID="acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30" exitCode=0 Sep 10 19:29:32.759502 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.759346 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:32.759502 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.759402 2584 scope.go:117] "RemoveContainer" containerID="acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30" Sep 10 19:29:32.768112 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.768092 2584 scope.go:117] "RemoveContainer" containerID="acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30" Sep 10 19:29:32.768435 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:29:32.768415 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30\": container with ID starting with acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30 not found: ID does not exist" containerID="acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30" Sep 10 19:29:32.768499 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.768444 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30"} err="failed to get container status \"acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30\": rpc error: code = NotFound desc = could not find container \"acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30\": container with ID starting with acb994d0862226dce6b3f45de971d2efb1321824807ebb0f2857225884752e30 not found: ID does not exist" Sep 10 19:29:32.772893 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:29:32.772868 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-86nlw\" is forbidden: User \"system:node:ip-10-0-134-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-64.ec2.internal' and this object" podUID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:32.819697 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.819655 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9" Sep 10 19:29:32.838193 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:32.838161 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-882n2\" (UniqueName: \"kubernetes.io/projected/fec21d8f-2b08-4a72-823c-dfbdd8d8d135-kube-api-access-882n2\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:29:33.101260 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:33.101234 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9"] Sep 10 19:29:33.103601 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:29:33.103571 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81f8072_fd40_4342_85cb_afe7956ddb62.slice/crio-7785c4b005e724af68486e23827db6dc1505c24f80364a14f05d379b2ddcb9cc WatchSource:0}: Error finding container 7785c4b005e724af68486e23827db6dc1505c24f80364a14f05d379b2ddcb9cc: Status 404 returned error can't find the container with id 7785c4b005e724af68486e23827db6dc1505c24f80364a14f05d379b2ddcb9cc Sep 10 19:29:33.765817 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:33.765775 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9" event={"ID":"b81f8072-fd40-4342-85cb-afe7956ddb62","Type":"ContainerStarted","Data":"4d8d80e2759d9acf8e62529140bb7acc0059ec395cd9877947fa0ef293ebb41d"} Sep 10 19:29:33.765817 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:33.765828 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9" event={"ID":"b81f8072-fd40-4342-85cb-afe7956ddb62","Type":"ContainerStarted","Data":"7785c4b005e724af68486e23827db6dc1505c24f80364a14f05d379b2ddcb9cc"} Sep 10 19:29:33.766345 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:33.765849 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9" Sep 10 19:29:33.768219 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:29:33.768191 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-86nlw\" is forbidden: User \"system:node:ip-10-0-134-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-64.ec2.internal' and this object" podUID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:33.798773 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:33.798703 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9" podStartSLOduration=1.798681496 podStartE2EDuration="1.798681496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:33.797908005 +0000 UTC m=+559.765720252" watchObservedRunningTime="2026-09-10 19:29:33.798681496 +0000 UTC m=+559.766493743" Sep 10 19:29:34.487673 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:34.487636 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" path="/var/lib/kubelet/pods/fec21d8f-2b08-4a72-823c-dfbdd8d8d135/volumes" Sep 10 19:29:34.488633 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:29:34.488610 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-86nlw\" is forbidden: User \"system:node:ip-10-0-134-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-64.ec2.internal' and this object" podUID="fec21d8f-2b08-4a72-823c-dfbdd8d8d135" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-86nlw" Sep 10 19:29:44.773192 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:29:44.773156 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zkph9" Sep 10 19:30:02.608196 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.608162 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x"] Sep 10 19:30:02.614307 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.614264 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.616630 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.616607 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8rxtk\"" Sep 10 19:30:02.625120 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.625094 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x"] Sep 10 19:30:02.665846 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.665805 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.665846 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.665865 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.666126 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.665931 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.666126 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.665971 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac470cae-838a-4175-bd45-be2ba7424eca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.666126 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.665997 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac470cae-838a-4175-bd45-be2ba7424eca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.666126 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.666027 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5kz\" (UniqueName: \"kubernetes.io/projected/ac470cae-838a-4175-bd45-be2ba7424eca-kube-api-access-6g5kz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.666126 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.666055 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.666126 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.666083 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac470cae-838a-4175-bd45-be2ba7424eca-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.666126 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.666122 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.766987 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.766947 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.766987 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.766984 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac470cae-838a-4175-bd45-be2ba7424eca-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767248 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767013 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767248 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767043 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767248 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767067 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767248 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767099 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767248 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767128 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac470cae-838a-4175-bd45-be2ba7424eca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767248 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767154 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac470cae-838a-4175-bd45-be2ba7424eca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767248 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767184 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5kz\" (UniqueName: \"kubernetes.io/projected/ac470cae-838a-4175-bd45-be2ba7424eca-kube-api-access-6g5kz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767693 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767502 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767693 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767581 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767693 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767614 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.767693 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.767678 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.768028 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.768003 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac470cae-838a-4175-bd45-be2ba7424eca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.769352 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.769330 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac470cae-838a-4175-bd45-be2ba7424eca-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.769812 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.769791 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac470cae-838a-4175-bd45-be2ba7424eca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.776590 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.776569 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5kz\" (UniqueName: \"kubernetes.io/projected/ac470cae-838a-4175-bd45-be2ba7424eca-kube-api-access-6g5kz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.777085 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.777068 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac470cae-838a-4175-bd45-be2ba7424eca-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7646x\" (UID: \"ac470cae-838a-4175-bd45-be2ba7424eca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:02.926152 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:02.926056 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:03.209406 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:03.209316 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x"] Sep 10 19:30:03.212266 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:30:03.212233 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac470cae_838a_4175_bd45_be2ba7424eca.slice/crio-fcff4257dc4b0abd535234abbd42215860a4d617c853552259c0f62d9dd050f8 WatchSource:0}: Error finding container fcff4257dc4b0abd535234abbd42215860a4d617c853552259c0f62d9dd050f8: Status 404 returned error can't find the container with id fcff4257dc4b0abd535234abbd42215860a4d617c853552259c0f62d9dd050f8 Sep 10 19:30:03.214389 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:03.214351 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:30:03.214464 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:03.214448 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:30:03.214519 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:03.214490 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:30:03.896107 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:03.896069 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" event={"ID":"ac470cae-838a-4175-bd45-be2ba7424eca","Type":"ContainerStarted","Data":"6d11c2e8ea24c94362463de9b5d2e0a4ee0826a58fbf6bfa190aad48705c5124"} Sep 10 19:30:03.896523 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:03.896135 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" event={"ID":"ac470cae-838a-4175-bd45-be2ba7424eca","Type":"ContainerStarted","Data":"fcff4257dc4b0abd535234abbd42215860a4d617c853552259c0f62d9dd050f8"} Sep 10 19:30:03.921313 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:03.921241 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" podStartSLOduration=1.921223875 podStartE2EDuration="1.921223875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:30:03.91898972 +0000 UTC m=+589.886801968" watchObservedRunningTime="2026-09-10 19:30:03.921223875 +0000 UTC m=+589.889036122" Sep 10 19:30:03.926807 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:03.926778 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:04.931561 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:04.931525 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:05.904767 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:05.904739 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:30:05.905595 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:30:05.905578 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:31:30.086103 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.086066 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg"] Sep 10 19:31:30.089436 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.089416 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" Sep 10 19:31:30.091742 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.091716 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-27clf\"" Sep 10 19:31:30.092725 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.092698 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 19:31:30.103360 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.103326 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg"] Sep 10 19:31:30.215610 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.215568 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c49cb9e-851e-4b1e-9ad1-3acb1c67038c-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-krgbg\" (UID: \"8c49cb9e-851e-4b1e-9ad1-3acb1c67038c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" Sep 10 19:31:30.215610 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.215614 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxfg9\" (UniqueName: \"kubernetes.io/projected/8c49cb9e-851e-4b1e-9ad1-3acb1c67038c-kube-api-access-qxfg9\") pod \"payload-pre-processing-59fbc47bf8-krgbg\" (UID: \"8c49cb9e-851e-4b1e-9ad1-3acb1c67038c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" Sep 10 19:31:30.317101 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.317061 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c49cb9e-851e-4b1e-9ad1-3acb1c67038c-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-krgbg\" (UID: \"8c49cb9e-851e-4b1e-9ad1-3acb1c67038c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" Sep 10 19:31:30.317101 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.317100 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxfg9\" (UniqueName: \"kubernetes.io/projected/8c49cb9e-851e-4b1e-9ad1-3acb1c67038c-kube-api-access-qxfg9\") pod \"payload-pre-processing-59fbc47bf8-krgbg\" (UID: \"8c49cb9e-851e-4b1e-9ad1-3acb1c67038c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" Sep 10 19:31:30.317823 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.317802 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c49cb9e-851e-4b1e-9ad1-3acb1c67038c-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-krgbg\" (UID: \"8c49cb9e-851e-4b1e-9ad1-3acb1c67038c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" Sep 10 19:31:30.326806 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.326768 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxfg9\" (UniqueName: \"kubernetes.io/projected/8c49cb9e-851e-4b1e-9ad1-3acb1c67038c-kube-api-access-qxfg9\") pod \"payload-pre-processing-59fbc47bf8-krgbg\" (UID: \"8c49cb9e-851e-4b1e-9ad1-3acb1c67038c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" Sep 10 19:31:30.400164 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.400070 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" Sep 10 19:31:30.688714 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:30.688686 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg"] Sep 10 19:31:30.690866 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:31:30.690835 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c49cb9e_851e_4b1e_9ad1_3acb1c67038c.slice/crio-12016a713055e4244710106c95f21180aeea8356d28ddd6adf319e0141dbdc28 WatchSource:0}: Error finding container 12016a713055e4244710106c95f21180aeea8356d28ddd6adf319e0141dbdc28: Status 404 returned error can't find the container with id 12016a713055e4244710106c95f21180aeea8356d28ddd6adf319e0141dbdc28 Sep 10 19:31:31.298020 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:31.297934 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" event={"ID":"8c49cb9e-851e-4b1e-9ad1-3acb1c67038c","Type":"ContainerStarted","Data":"12016a713055e4244710106c95f21180aeea8356d28ddd6adf319e0141dbdc28"} Sep 10 19:31:33.308855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:33.308818 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" event={"ID":"8c49cb9e-851e-4b1e-9ad1-3acb1c67038c","Type":"ContainerStarted","Data":"1855b66bbd59eb24ab9f39937371481a60929f36c625fdbac7a266751d11d64a"} Sep 10 19:31:33.309250 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:33.308910 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" Sep 10 19:31:33.328193 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:33.328116 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" podStartSLOduration=1.451084478 podStartE2EDuration="3.328080722s" totalImagesPullingTime="1.876996244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:31:33.325752607 +0000 UTC m=+679.293564855" watchObservedRunningTime="2026-09-10 19:31:33.328080722 +0000 UTC m=+679.295892970" Sep 10 19:31:40.349555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:40.349517 2584 generic.go:356] "Generic (PLEG): container finished" podID="ac470cae-838a-4175-bd45-be2ba7424eca" containerID="6d11c2e8ea24c94362463de9b5d2e0a4ee0826a58fbf6bfa190aad48705c5124" exitCode=0 Sep 10 19:31:40.349971 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:40.349590 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" event={"ID":"ac470cae-838a-4175-bd45-be2ba7424eca","Type":"ContainerDied","Data":"6d11c2e8ea24c94362463de9b5d2e0a4ee0826a58fbf6bfa190aad48705c5124"} Sep 10 19:31:40.350013 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:40.349996 2584 scope.go:117] "RemoveContainer" containerID="6d11c2e8ea24c94362463de9b5d2e0a4ee0826a58fbf6bfa190aad48705c5124" Sep 10 19:31:40.351074 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:40.351047 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:31:40.351146 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:40.351119 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:31:40.351188 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:40.351150 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:31:41.355921 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:41.355887 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" event={"ID":"ac470cae-838a-4175-bd45-be2ba7424eca","Type":"ContainerStarted","Data":"36621e0403db22948e9a5a3bc4280ba3c2495bae695ecb061a8fc3f20f563b0c"} Sep 10 19:31:41.927152 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:41.927062 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:31:41.927503 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:41.927474 2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7646x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Sep 10 19:31:41.927678 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:41.927647 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" podUID="ac470cae-838a-4175-bd45-be2ba7424eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Sep 10 19:31:42.927052 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:42.927009 2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7646x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Sep 10 19:31:42.928065 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:42.927629 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" podUID="ac470cae-838a-4175-bd45-be2ba7424eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Sep 10 19:31:43.927313 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:43.926808 2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7646x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Sep 10 19:31:43.927313 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:43.926877 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" podUID="ac470cae-838a-4175-bd45-be2ba7424eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Sep 10 19:31:44.316514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:44.316472 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-krgbg" Sep 10 19:31:44.926921 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:44.926874 2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7646x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Sep 10 19:31:44.927227 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:44.927194 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" podUID="ac470cae-838a-4175-bd45-be2ba7424eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Sep 10 19:31:45.927408 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:45.927364 2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7646x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Sep 10 19:31:45.928011 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:45.927976 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" podUID="ac470cae-838a-4175-bd45-be2ba7424eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Sep 10 19:31:46.930344 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:46.930309 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:31:46.930843 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:46.930699 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:31:46.931308 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:46.931271 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7646x" Sep 10 19:31:56.269541 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.269504 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld"] Sep 10 19:31:56.272317 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.272291 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.274741 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.274716 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 19:31:56.274882 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.274725 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xklpr\"" Sep 10 19:31:56.275581 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.275560 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 19:31:56.275678 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.275616 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 19:31:56.286700 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.286676 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld"] Sep 10 19:31:56.363491 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.363451 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.363692 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.363518 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8433d3fe-d47b-46c1-8811-28dc17a1325b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.363692 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.363553 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.363692 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.363597 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.363692 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.363644 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fmv\" (UniqueName: \"kubernetes.io/projected/8433d3fe-d47b-46c1-8811-28dc17a1325b-kube-api-access-p6fmv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.363861 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.363713 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.464668 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.464617 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.464870 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.464697 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.464870 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.464724 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fmv\" (UniqueName: \"kubernetes.io/projected/8433d3fe-d47b-46c1-8811-28dc17a1325b-kube-api-access-p6fmv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.464870 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.464772 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.464870 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.464807 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.464870 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.464859 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8433d3fe-d47b-46c1-8811-28dc17a1325b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.465127 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.465111 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.465188 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.465159 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.465235 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.465200 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.467305 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.467263 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8433d3fe-d47b-46c1-8811-28dc17a1325b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.467523 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.467504 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8433d3fe-d47b-46c1-8811-28dc17a1325b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.473063 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.473039 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fmv\" (UniqueName: \"kubernetes.io/projected/8433d3fe-d47b-46c1-8811-28dc17a1325b-kube-api-access-p6fmv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxnld\" (UID: \"8433d3fe-d47b-46c1-8811-28dc17a1325b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.582908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.582860 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:31:56.855316 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:56.855206 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld"] Sep 10 19:31:56.859018 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:31:56.858988 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8433d3fe_d47b_46c1_8811_28dc17a1325b.slice/crio-ac9152fb384e3b38f49ee371201682c6013ae68beffb6e837784ec07ebaad68f WatchSource:0}: Error finding container ac9152fb384e3b38f49ee371201682c6013ae68beffb6e837784ec07ebaad68f: Status 404 returned error can't find the container with id ac9152fb384e3b38f49ee371201682c6013ae68beffb6e837784ec07ebaad68f Sep 10 19:31:57.432127 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:57.432082 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" event={"ID":"8433d3fe-d47b-46c1-8811-28dc17a1325b","Type":"ContainerStarted","Data":"ac9152fb384e3b38f49ee371201682c6013ae68beffb6e837784ec07ebaad68f"} Sep 10 19:31:58.438378 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:58.438339 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" event={"ID":"8433d3fe-d47b-46c1-8811-28dc17a1325b","Type":"ContainerStarted","Data":"6981cd8e710caa338706b15b1493ce362391524697c3711203d02b033f7cc51d"} Sep 10 19:31:58.466852 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:31:58.466795 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" podStartSLOduration=1.4613575380000001 podStartE2EDuration="2.4667765s" totalImagesPullingTime="1.005418962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:31:58.464984373 +0000 UTC m=+704.432796620" watchObservedRunningTime="2026-09-10 19:31:58.4667765 +0000 UTC m=+704.434588776" Sep 10 19:32:06.583104 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:06.583058 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:32:06.583541 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:06.583126 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:32:06.595385 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:06.595358 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:32:07.491466 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:07.491436 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxnld" Sep 10 19:32:29.873370 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:29.873329 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b"] Sep 10 19:32:29.876531 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:29.876505 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:29.878990 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:29.878968 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 10 19:32:29.889348 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:29.889320 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b"] Sep 10 19:32:29.952683 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:29.952639 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad91ab45-5705-4499-9a03-e3ca53cdb107-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:29.952683 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:29.952684 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:29.952889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:29.952783 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq49s\" (UniqueName: \"kubernetes.io/projected/ad91ab45-5705-4499-9a03-e3ca53cdb107-kube-api-access-mq49s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:29.952889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:29.952860 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:29.952889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:29.952884 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:29.952984 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:29.952906 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.053624 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.053582 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.053624 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.053630 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.053878 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.053647 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.053878 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.053669 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad91ab45-5705-4499-9a03-e3ca53cdb107-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.053878 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.053689 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.053878 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.053735 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq49s\" (UniqueName: \"kubernetes.io/projected/ad91ab45-5705-4499-9a03-e3ca53cdb107-kube-api-access-mq49s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.054092 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.054028 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.054141 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.054082 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.054141 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.054130 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.056194 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.056160 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad91ab45-5705-4499-9a03-e3ca53cdb107-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.056422 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.056403 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad91ab45-5705-4499-9a03-e3ca53cdb107-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.066335 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.066309 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq49s\" (UniqueName: \"kubernetes.io/projected/ad91ab45-5705-4499-9a03-e3ca53cdb107-kube-api-access-mq49s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b\" (UID: \"ad91ab45-5705-4499-9a03-e3ca53cdb107\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.186246 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.186143 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:30.467680 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.467576 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b"] Sep 10 19:32:30.472744 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:32:30.472712 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad91ab45_5705_4499_9a03_e3ca53cdb107.slice/crio-58021f718412252da51ed3723ae20b65c5f44db27d94a00566c260ecb292bc46 WatchSource:0}: Error finding container 58021f718412252da51ed3723ae20b65c5f44db27d94a00566c260ecb292bc46: Status 404 returned error can't find the container with id 58021f718412252da51ed3723ae20b65c5f44db27d94a00566c260ecb292bc46 Sep 10 19:32:30.473986 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.473962 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:32:30.592205 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:30.592165 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" event={"ID":"ad91ab45-5705-4499-9a03-e3ca53cdb107","Type":"ContainerStarted","Data":"58021f718412252da51ed3723ae20b65c5f44db27d94a00566c260ecb292bc46"} Sep 10 19:32:31.598531 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:31.598496 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" event={"ID":"ad91ab45-5705-4499-9a03-e3ca53cdb107","Type":"ContainerStarted","Data":"ce3074e6d3a28263d6a28235d3403128d6b8ddaa468d5ed76e2ef498865e3079"} Sep 10 19:32:31.619813 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:31.619755 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" podStartSLOduration=2.410123282 podStartE2EDuration="2.619737981s" totalImagesPullingTime="209.614699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:31.619219264 +0000 UTC m=+737.587031512" watchObservedRunningTime="2026-09-10 19:32:31.619737981 +0000 UTC m=+737.587550228" Sep 10 19:32:39.481693 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.481657 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4"] Sep 10 19:32:39.484170 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.484151 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.486581 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.486562 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 10 19:32:39.498018 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.497982 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4"] Sep 10 19:32:39.538364 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.538325 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.538555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.538383 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.538555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.538446 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.538555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.538509 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnrww\" (UniqueName: \"kubernetes.io/projected/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-kube-api-access-mnrww\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.538555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.538550 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.538711 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.538610 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.639048 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.639011 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.639048 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.639050 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrww\" (UniqueName: \"kubernetes.io/projected/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-kube-api-access-mnrww\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.639353 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.639075 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.639353 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.639122 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.639353 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.639200 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.639353 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.639239 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.639621 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.639597 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.639675 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.639633 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.639675 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.639660 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.641453 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.641432 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.641606 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.641589 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.648853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.648817 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrww\" (UniqueName: \"kubernetes.io/projected/f890f0f7-cdbc-4ff3-870b-b9479ac0261e-kube-api-access-mnrww\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xphq4\" (UID: \"f890f0f7-cdbc-4ff3-870b-b9479ac0261e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:39.793530 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:39.793434 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:40.068784 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:40.068719 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4"] Sep 10 19:32:40.071432 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:32:40.071399 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf890f0f7_cdbc_4ff3_870b_b9479ac0261e.slice/crio-aa84d21388f0e1b76fa247a3c98c9a16d5cc41b5bc40a453b338d5e898e240dc WatchSource:0}: Error finding container aa84d21388f0e1b76fa247a3c98c9a16d5cc41b5bc40a453b338d5e898e240dc: Status 404 returned error can't find the container with id aa84d21388f0e1b76fa247a3c98c9a16d5cc41b5bc40a453b338d5e898e240dc Sep 10 19:32:40.186659 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:40.186598 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:40.186848 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:40.186722 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:40.199091 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:40.199061 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:40.640550 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:40.640455 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" event={"ID":"f890f0f7-cdbc-4ff3-870b-b9479ac0261e","Type":"ContainerStarted","Data":"0937a413b2948dd094adf995b61811842be3362d8b9cc2c255aa1ca48190168d"} Sep 10 19:32:40.640550 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:40.640503 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" event={"ID":"f890f0f7-cdbc-4ff3-870b-b9479ac0261e","Type":"ContainerStarted","Data":"aa84d21388f0e1b76fa247a3c98c9a16d5cc41b5bc40a453b338d5e898e240dc"} Sep 10 19:32:40.652083 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:40.652049 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b" Sep 10 19:32:40.659876 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:40.659825 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" podStartSLOduration=1.457635474 podStartE2EDuration="1.659808706s" totalImagesPullingTime="202.173232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:40.657983614 +0000 UTC m=+746.625795862" watchObservedRunningTime="2026-09-10 19:32:40.659808706 +0000 UTC m=+746.627620953" Sep 10 19:32:49.794370 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:49.794309 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:49.794370 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:49.794383 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:49.807001 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:49.806962 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:32:50.698216 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:32:50.698183 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xphq4" Sep 10 19:34:19.158208 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.158171 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk"] Sep 10 19:34:19.161580 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.161557 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.164338 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.164315 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3a1590-openshift-default-dockercfg-7gv4r\"" Sep 10 19:34:19.177551 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.177519 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk"] Sep 10 19:34:19.216298 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.216253 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-envoy\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.216480 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.216310 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6440fe6-251e-4ea5-94be-4f25951985ab-istiod-ca-cert\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.216480 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.216342 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-podinfo\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.216480 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.216392 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-socket\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.216480 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.216434 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-token\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.216480 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.216456 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddj8t\" (UniqueName: \"kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-kube-api-access-ddj8t\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.216686 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.216484 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-data\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.216686 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.216512 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-credential-socket\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.216686 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.216569 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-certs\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.317831 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.317794 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddj8t\" (UniqueName: \"kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-kube-api-access-ddj8t\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.318034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.317844 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-data\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.318034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.317876 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-credential-socket\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.318034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.317905 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-certs\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.318034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.317934 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-envoy\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.318034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.317967 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6440fe6-251e-4ea5-94be-4f25951985ab-istiod-ca-cert\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.318034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.317994 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-podinfo\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.318034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.318033 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-socket\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.318412 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.318085 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-token\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.318467 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.318415 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-data\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.318963 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.318881 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-socket\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.319100 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.319029 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6440fe6-251e-4ea5-94be-4f25951985ab-istiod-ca-cert\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.319100 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.319091 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-certs\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.319670 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.319641 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-credential-socket\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.321449 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.320983 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-envoy\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.321449 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.321351 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-podinfo\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.330251 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.330219 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-token\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.330401 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.330248 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddj8t\" (UniqueName: \"kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-kube-api-access-ddj8t\") pod \"e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.472552 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.472458 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:19.788250 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.788168 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk"] Sep 10 19:34:19.794579 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:34:19.794543 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6440fe6_251e_4ea5_94be_4f25951985ab.slice/crio-dec096185736d2896a7084ce0492111c6892aa13e48408256979af445059705b WatchSource:0}: Error finding container dec096185736d2896a7084ce0492111c6892aa13e48408256979af445059705b: Status 404 returned error can't find the container with id dec096185736d2896a7084ce0492111c6892aa13e48408256979af445059705b Sep 10 19:34:19.802136 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.797821 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:34:19.802136 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.797992 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:34:19.802136 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:19.798081 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:34:20.102600 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:20.102555 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" event={"ID":"e6440fe6-251e-4ea5-94be-4f25951985ab","Type":"ContainerStarted","Data":"a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd"} Sep 10 19:34:20.102776 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:20.102648 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" event={"ID":"e6440fe6-251e-4ea5-94be-4f25951985ab","Type":"ContainerStarted","Data":"dec096185736d2896a7084ce0492111c6892aa13e48408256979af445059705b"} Sep 10 19:34:20.127517 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:20.127452 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" podStartSLOduration=1.127435508 podStartE2EDuration="1.127435508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:20.125907819 +0000 UTC m=+846.093720067" watchObservedRunningTime="2026-09-10 19:34:20.127435508 +0000 UTC m=+846.095247754" Sep 10 19:34:20.473313 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:20.473191 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:20.478380 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:20.478351 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:21.107955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:21.107922 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:21.108897 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:21.108872 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:24.167006 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.166963 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt"] Sep 10 19:34:24.170778 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.170752 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:34:24.172961 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.172932 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3a1590\"" Sep 10 19:34:24.173098 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.172996 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3a1590-dockercfg-57w5c\"" Sep 10 19:34:24.181131 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.181102 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt"] Sep 10 19:34:24.182830 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.182806 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb"] Sep 10 19:34:24.186436 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.186413 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:34:24.196273 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.196248 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb"] Sep 10 19:34:24.266394 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.266355 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqrm\" (UniqueName: \"kubernetes.io/projected/b1d8a473-18f0-4235-bcf3-570f227b5df7-kube-api-access-6wqrm\") pod \"payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt\" (UID: \"b1d8a473-18f0-4235-bcf3-570f227b5df7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:34:24.266564 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.266416 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d8a473-18f0-4235-bcf3-570f227b5df7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt\" (UID: \"b1d8a473-18f0-4235-bcf3-570f227b5df7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:34:24.266564 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.266527 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fdbb8b-66f5-4d74-a704-6581f1771709-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb\" (UID: \"f4fdbb8b-66f5-4d74-a704-6581f1771709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:34:24.266662 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.266614 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxzq\" (UniqueName: \"kubernetes.io/projected/f4fdbb8b-66f5-4d74-a704-6581f1771709-kube-api-access-tbxzq\") pod \"payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb\" (UID: \"f4fdbb8b-66f5-4d74-a704-6581f1771709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:34:24.368090 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.368045 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxzq\" (UniqueName: \"kubernetes.io/projected/f4fdbb8b-66f5-4d74-a704-6581f1771709-kube-api-access-tbxzq\") pod \"payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb\" (UID: \"f4fdbb8b-66f5-4d74-a704-6581f1771709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:34:24.368416 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.368142 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqrm\" (UniqueName: \"kubernetes.io/projected/b1d8a473-18f0-4235-bcf3-570f227b5df7-kube-api-access-6wqrm\") pod \"payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt\" (UID: \"b1d8a473-18f0-4235-bcf3-570f227b5df7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:34:24.368416 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.368187 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d8a473-18f0-4235-bcf3-570f227b5df7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt\" (UID: \"b1d8a473-18f0-4235-bcf3-570f227b5df7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:34:24.368416 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.368230 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fdbb8b-66f5-4d74-a704-6581f1771709-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb\" (UID: \"f4fdbb8b-66f5-4d74-a704-6581f1771709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:34:24.369019 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.368993 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fdbb8b-66f5-4d74-a704-6581f1771709-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb\" (UID: \"f4fdbb8b-66f5-4d74-a704-6581f1771709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:34:24.369110 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.369082 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d8a473-18f0-4235-bcf3-570f227b5df7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt\" (UID: \"b1d8a473-18f0-4235-bcf3-570f227b5df7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:34:24.376436 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.376405 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxzq\" (UniqueName: \"kubernetes.io/projected/f4fdbb8b-66f5-4d74-a704-6581f1771709-kube-api-access-tbxzq\") pod \"payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb\" (UID: \"f4fdbb8b-66f5-4d74-a704-6581f1771709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:34:24.376635 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.376619 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqrm\" (UniqueName: \"kubernetes.io/projected/b1d8a473-18f0-4235-bcf3-570f227b5df7-kube-api-access-6wqrm\") pod \"payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt\" (UID: \"b1d8a473-18f0-4235-bcf3-570f227b5df7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:34:24.482171 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.482073 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:34:24.499026 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.498986 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:34:24.757668 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.757640 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt"] Sep 10 19:34:24.759692 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:34:24.759668 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d8a473_18f0_4235_bcf3_570f227b5df7.slice/crio-766d8a4aacde8ce61d6616787df2ff72e5d6ce147efeacd35f97f1181edb2175 WatchSource:0}: Error finding container 766d8a4aacde8ce61d6616787df2ff72e5d6ce147efeacd35f97f1181edb2175: Status 404 returned error can't find the container with id 766d8a4aacde8ce61d6616787df2ff72e5d6ce147efeacd35f97f1181edb2175 Sep 10 19:34:24.791060 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:24.791030 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb"] Sep 10 19:34:24.802939 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:34:24.802900 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fdbb8b_66f5_4d74_a704_6581f1771709.slice/crio-7a7e49c21a8931b2c85dc4044ee3c19bfe34a1cb47028b5764936742b92f6bf0 WatchSource:0}: Error finding container 7a7e49c21a8931b2c85dc4044ee3c19bfe34a1cb47028b5764936742b92f6bf0: Status 404 returned error can't find the container with id 7a7e49c21a8931b2c85dc4044ee3c19bfe34a1cb47028b5764936742b92f6bf0 Sep 10 19:34:25.130351 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:25.130312 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" event={"ID":"f4fdbb8b-66f5-4d74-a704-6581f1771709","Type":"ContainerStarted","Data":"04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba"} Sep 10 19:34:25.130543 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:25.130379 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" event={"ID":"f4fdbb8b-66f5-4d74-a704-6581f1771709","Type":"ContainerStarted","Data":"7a7e49c21a8931b2c85dc4044ee3c19bfe34a1cb47028b5764936742b92f6bf0"} Sep 10 19:34:25.130543 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:25.130480 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:34:25.132987 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:25.132955 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" event={"ID":"b1d8a473-18f0-4235-bcf3-570f227b5df7","Type":"ContainerStarted","Data":"11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8"} Sep 10 19:34:25.133155 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:25.133005 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" event={"ID":"b1d8a473-18f0-4235-bcf3-570f227b5df7","Type":"ContainerStarted","Data":"766d8a4aacde8ce61d6616787df2ff72e5d6ce147efeacd35f97f1181edb2175"} Sep 10 19:34:25.133155 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:25.133045 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:34:25.147558 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:25.147498 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" podStartSLOduration=1.1474802610000001 podStartE2EDuration="1.147480261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:25.145940511 +0000 UTC m=+851.113752761" watchObservedRunningTime="2026-09-10 19:34:25.147480261 +0000 UTC m=+851.115292507" Sep 10 19:34:25.164451 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:25.164382 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" podStartSLOduration=1.164361209 podStartE2EDuration="1.164361209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:25.162375784 +0000 UTC m=+851.130188034" watchObservedRunningTime="2026-09-10 19:34:25.164361209 +0000 UTC m=+851.132173460" Sep 10 19:34:35.296178 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.296140 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn"] Sep 10 19:34:35.299661 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.299643 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:34:35.306038 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.306014 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f7189cc9-dockercfg-b6zpf\"" Sep 10 19:34:35.306445 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.306425 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f7189cc9\"" Sep 10 19:34:35.311945 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.311919 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn"] Sep 10 19:34:35.345734 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.345665 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t"] Sep 10 19:34:35.349821 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.349784 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t"] Sep 10 19:34:35.349994 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.349898 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:34:35.374753 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.374718 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t27p\" (UniqueName: \"kubernetes.io/projected/5d7dc494-c869-4776-99c6-125374169366-kube-api-access-5t27p\") pod \"payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn\" (UID: \"5d7dc494-c869-4776-99c6-125374169366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:34:35.374927 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.374777 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7dc494-c869-4776-99c6-125374169366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn\" (UID: \"5d7dc494-c869-4776-99c6-125374169366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:34:35.475777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.475734 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t\" (UID: \"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f\") " pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:34:35.475969 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.475796 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t27p\" (UniqueName: \"kubernetes.io/projected/5d7dc494-c869-4776-99c6-125374169366-kube-api-access-5t27p\") pod \"payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn\" (UID: \"5d7dc494-c869-4776-99c6-125374169366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:34:35.475969 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.475859 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7ps\" (UniqueName: \"kubernetes.io/projected/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-kube-api-access-7z7ps\") pod \"payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t\" (UID: \"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f\") " pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:34:35.475969 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.475927 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7dc494-c869-4776-99c6-125374169366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn\" (UID: \"5d7dc494-c869-4776-99c6-125374169366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:34:35.476776 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.476759 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7dc494-c869-4776-99c6-125374169366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn\" (UID: \"5d7dc494-c869-4776-99c6-125374169366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:34:35.492544 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.492514 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t27p\" (UniqueName: \"kubernetes.io/projected/5d7dc494-c869-4776-99c6-125374169366-kube-api-access-5t27p\") pod \"payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn\" (UID: \"5d7dc494-c869-4776-99c6-125374169366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:34:35.577120 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.577078 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7ps\" (UniqueName: \"kubernetes.io/projected/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-kube-api-access-7z7ps\") pod \"payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t\" (UID: \"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f\") " pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:34:35.577626 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.577377 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t\" (UID: \"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f\") " pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:34:35.578137 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.578110 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t\" (UID: \"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f\") " pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:34:35.587073 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.587045 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7ps\" (UniqueName: \"kubernetes.io/projected/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-kube-api-access-7z7ps\") pod \"payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t\" (UID: \"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f\") " pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:34:35.610740 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.610706 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:34:35.660506 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.660468 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:34:35.921789 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.921757 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn"] Sep 10 19:34:35.924360 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:34:35.924329 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7dc494_c869_4776_99c6_125374169366.slice/crio-aea067b4a2313ce58bde3964f6739b1f167e39c5db04c411951a889a4034c485 WatchSource:0}: Error finding container aea067b4a2313ce58bde3964f6739b1f167e39c5db04c411951a889a4034c485: Status 404 returned error can't find the container with id aea067b4a2313ce58bde3964f6739b1f167e39c5db04c411951a889a4034c485 Sep 10 19:34:35.965907 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:35.965878 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t"] Sep 10 19:34:35.967621 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:34:35.967584 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b3408d_4991_4d9d_8f5a_833e05ee8c0f.slice/crio-a9d0b3695bde8f22e48fc0050c7c87023fcfafd947ba7790e7a3987f304b3f2e WatchSource:0}: Error finding container a9d0b3695bde8f22e48fc0050c7c87023fcfafd947ba7790e7a3987f304b3f2e: Status 404 returned error can't find the container with id a9d0b3695bde8f22e48fc0050c7c87023fcfafd947ba7790e7a3987f304b3f2e Sep 10 19:34:36.140371 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.140338 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:34:36.140555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.140431 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:34:36.191749 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.191660 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" event={"ID":"5d7dc494-c869-4776-99c6-125374169366","Type":"ContainerStarted","Data":"e38382bfdc4cebf54b2889c6401555831abd38c92172bfacc440068f0381e44a"} Sep 10 19:34:36.192739 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.192598 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:34:36.193270 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.193248 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" event={"ID":"5d7dc494-c869-4776-99c6-125374169366","Type":"ContainerStarted","Data":"aea067b4a2313ce58bde3964f6739b1f167e39c5db04c411951a889a4034c485"} Sep 10 19:34:36.194444 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.194423 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" event={"ID":"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f","Type":"ContainerStarted","Data":"0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05"} Sep 10 19:34:36.194968 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.194944 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" event={"ID":"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f","Type":"ContainerStarted","Data":"a9d0b3695bde8f22e48fc0050c7c87023fcfafd947ba7790e7a3987f304b3f2e"} Sep 10 19:34:36.195183 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.195153 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:34:36.213689 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.213210 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" podStartSLOduration=1.213189409 podStartE2EDuration="1.213189409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:36.212930803 +0000 UTC m=+862.180743051" watchObservedRunningTime="2026-09-10 19:34:36.213189409 +0000 UTC m=+862.181001661" Sep 10 19:34:36.237150 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.237081 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" podStartSLOduration=1.237040722 podStartE2EDuration="1.237040722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:36.236456546 +0000 UTC m=+862.204268786" watchObservedRunningTime="2026-09-10 19:34:36.237040722 +0000 UTC m=+862.204852970" Sep 10 19:34:36.450929 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.450839 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn"] Sep 10 19:34:36.455467 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.455441 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:34:36.458165 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.458141 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-18c3c3-dockercfg-4gm85\"" Sep 10 19:34:36.458377 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.458141 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-18c3c3\"" Sep 10 19:34:36.464378 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.464353 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn"] Sep 10 19:34:36.587892 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.587847 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnln\" (UniqueName: \"kubernetes.io/projected/941884eb-043a-411b-88eb-ebc67ec1f7fc-kube-api-access-5dnln\") pod \"payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn\" (UID: \"941884eb-043a-411b-88eb-ebc67ec1f7fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:34:36.588089 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.587965 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941884eb-043a-411b-88eb-ebc67ec1f7fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn\" (UID: \"941884eb-043a-411b-88eb-ebc67ec1f7fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:34:36.688649 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.688604 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnln\" (UniqueName: \"kubernetes.io/projected/941884eb-043a-411b-88eb-ebc67ec1f7fc-kube-api-access-5dnln\") pod \"payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn\" (UID: \"941884eb-043a-411b-88eb-ebc67ec1f7fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:34:36.688649 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.688668 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941884eb-043a-411b-88eb-ebc67ec1f7fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn\" (UID: \"941884eb-043a-411b-88eb-ebc67ec1f7fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:34:36.689459 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.689433 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941884eb-043a-411b-88eb-ebc67ec1f7fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn\" (UID: \"941884eb-043a-411b-88eb-ebc67ec1f7fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:34:36.698870 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.698837 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnln\" (UniqueName: \"kubernetes.io/projected/941884eb-043a-411b-88eb-ebc67ec1f7fc-kube-api-access-5dnln\") pod \"payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn\" (UID: \"941884eb-043a-411b-88eb-ebc67ec1f7fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:34:36.772571 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:36.772467 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:34:37.069694 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:37.069639 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn"] Sep 10 19:34:37.073518 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:34:37.073479 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941884eb_043a_411b_88eb_ebc67ec1f7fc.slice/crio-dfb1c84f66a7c11a33c4640f98af9f31173617454c8d0235bedc057ca6339ded WatchSource:0}: Error finding container dfb1c84f66a7c11a33c4640f98af9f31173617454c8d0235bedc057ca6339ded: Status 404 returned error can't find the container with id dfb1c84f66a7c11a33c4640f98af9f31173617454c8d0235bedc057ca6339ded Sep 10 19:34:37.148457 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:37.148419 2584 patch_prober.go:28] interesting pod/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 19:34:37.148608 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:37.148481 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 19:34:37.201691 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:37.201566 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" event={"ID":"941884eb-043a-411b-88eb-ebc67ec1f7fc","Type":"ContainerStarted","Data":"3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3"} Sep 10 19:34:37.201691 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:37.201630 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" event={"ID":"941884eb-043a-411b-88eb-ebc67ec1f7fc","Type":"ContainerStarted","Data":"dfb1c84f66a7c11a33c4640f98af9f31173617454c8d0235bedc057ca6339ded"} Sep 10 19:34:37.201954 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:37.201800 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:34:37.221166 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:37.221091 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" podStartSLOduration=1.221065629 podStartE2EDuration="1.221065629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:37.218206692 +0000 UTC m=+863.186018944" watchObservedRunningTime="2026-09-10 19:34:37.221065629 +0000 UTC m=+863.188877885" Sep 10 19:34:39.213306 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:39.213200 2584 generic.go:356] "Generic (PLEG): container finished" podID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerID="a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd" exitCode=0 Sep 10 19:34:39.213306 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:39.213271 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" event={"ID":"e6440fe6-251e-4ea5-94be-4f25951985ab","Type":"ContainerDied","Data":"a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd"} Sep 10 19:34:39.213741 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:39.213683 2584 scope.go:117] "RemoveContainer" containerID="a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd" Sep 10 19:34:39.214753 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:39.214721 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:34:39.214839 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:39.214815 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:34:39.214890 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:39.214848 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:34:40.224869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:40.224828 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" event={"ID":"e6440fe6-251e-4ea5-94be-4f25951985ab","Type":"ContainerStarted","Data":"396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955"} Sep 10 19:34:40.473116 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:40.473068 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:40.473461 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:40.473439 2584 patch_prober.go:28] interesting pod/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 10 19:34:40.473523 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:40.473485 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 10 19:34:41.472889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:41.472848 2584 patch_prober.go:28] interesting pod/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 10 19:34:41.473299 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:41.472910 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 10 19:34:42.473725 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:42.473689 2584 patch_prober.go:28] interesting pod/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 10 19:34:42.474152 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:42.473746 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 10 19:34:43.473079 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:43.473034 2584 patch_prober.go:28] interesting pod/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 10 19:34:43.473317 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:43.473099 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 10 19:34:44.472894 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:44.472856 2584 patch_prober.go:28] interesting pod/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 10 19:34:44.473333 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:44.472914 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 10 19:34:45.476784 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:45.476753 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:45.477185 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:45.477016 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:45.477741 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:45.477722 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:34:48.209686 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:48.209643 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:34:48.210074 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:48.209739 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:34:48.210074 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:48.209877 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:34:53.077030 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.076992 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj"] Sep 10 19:34:53.080729 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.080710 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:34:53.088227 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.088202 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a037c8-dockercfg-46gc6\"" Sep 10 19:34:53.088530 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.088514 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a037c8\"" Sep 10 19:34:53.126452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.126405 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj"] Sep 10 19:34:53.249327 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.249267 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6whhh\" (UniqueName: \"kubernetes.io/projected/e322e040-cc76-4925-8171-ba15d9567bc5-kube-api-access-6whhh\") pod \"payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj\" (UID: \"e322e040-cc76-4925-8171-ba15d9567bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:34:53.249487 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.249413 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e322e040-cc76-4925-8171-ba15d9567bc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj\" (UID: \"e322e040-cc76-4925-8171-ba15d9567bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:34:53.350611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.350504 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e322e040-cc76-4925-8171-ba15d9567bc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj\" (UID: \"e322e040-cc76-4925-8171-ba15d9567bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:34:53.350611 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.350571 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6whhh\" (UniqueName: \"kubernetes.io/projected/e322e040-cc76-4925-8171-ba15d9567bc5-kube-api-access-6whhh\") pod \"payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj\" (UID: \"e322e040-cc76-4925-8171-ba15d9567bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:34:53.351236 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.351212 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e322e040-cc76-4925-8171-ba15d9567bc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj\" (UID: \"e322e040-cc76-4925-8171-ba15d9567bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:34:53.362859 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.362829 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6whhh\" (UniqueName: \"kubernetes.io/projected/e322e040-cc76-4925-8171-ba15d9567bc5-kube-api-access-6whhh\") pod \"payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj\" (UID: \"e322e040-cc76-4925-8171-ba15d9567bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:34:53.391929 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.391887 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:34:53.670798 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:53.670719 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj"] Sep 10 19:34:53.676735 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:34:53.676705 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode322e040_cc76_4925_8171_ba15d9567bc5.slice/crio-9a7c17f8d8463f11f881b5fd689984fa29b2382f09b9884cad2bd9a6d46e7e35 WatchSource:0}: Error finding container 9a7c17f8d8463f11f881b5fd689984fa29b2382f09b9884cad2bd9a6d46e7e35: Status 404 returned error can't find the container with id 9a7c17f8d8463f11f881b5fd689984fa29b2382f09b9884cad2bd9a6d46e7e35 Sep 10 19:34:54.307840 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:54.307795 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" event={"ID":"e322e040-cc76-4925-8171-ba15d9567bc5","Type":"ContainerStarted","Data":"9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1"} Sep 10 19:34:54.308332 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:54.307868 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:34:54.308332 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:54.307887 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" event={"ID":"e322e040-cc76-4925-8171-ba15d9567bc5","Type":"ContainerStarted","Data":"9a7c17f8d8463f11f881b5fd689984fa29b2382f09b9884cad2bd9a6d46e7e35"} Sep 10 19:34:54.326124 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:34:54.326062 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" podStartSLOduration=1.326044261 podStartE2EDuration="1.326044261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:54.324487045 +0000 UTC m=+880.292299309" watchObservedRunningTime="2026-09-10 19:34:54.326044261 +0000 UTC m=+880.293856508" Sep 10 19:35:05.316434 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:05.316390 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:35:15.118720 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.118679 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t"] Sep 10 19:35:15.119130 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.118953 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" podUID="f8b3408d-4991-4d9d-8f5a-833e05ee8c0f" containerName="payload-processing" containerID="cri-o://0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05" gracePeriod=30 Sep 10 19:35:15.131870 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.131835 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn"] Sep 10 19:35:15.132113 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.132085 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" podUID="5d7dc494-c869-4776-99c6-125374169366" containerName="payload-pre-processing" containerID="cri-o://e38382bfdc4cebf54b2889c6401555831abd38c92172bfacc440068f0381e44a" gracePeriod=30 Sep 10 19:35:15.415188 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.414475 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:35:15.418036 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.418007 2584 generic.go:356] "Generic (PLEG): container finished" podID="5d7dc494-c869-4776-99c6-125374169366" containerID="e38382bfdc4cebf54b2889c6401555831abd38c92172bfacc440068f0381e44a" exitCode=0 Sep 10 19:35:15.418172 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.418081 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" event={"ID":"5d7dc494-c869-4776-99c6-125374169366","Type":"ContainerDied","Data":"e38382bfdc4cebf54b2889c6401555831abd38c92172bfacc440068f0381e44a"} Sep 10 19:35:15.418172 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.418133 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" event={"ID":"5d7dc494-c869-4776-99c6-125374169366","Type":"ContainerDied","Data":"aea067b4a2313ce58bde3964f6739b1f167e39c5db04c411951a889a4034c485"} Sep 10 19:35:15.418172 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.418157 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aea067b4a2313ce58bde3964f6739b1f167e39c5db04c411951a889a4034c485" Sep 10 19:35:15.419040 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.419003 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:35:15.419569 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.419550 2584 generic.go:356] "Generic (PLEG): container finished" podID="f8b3408d-4991-4d9d-8f5a-833e05ee8c0f" containerID="0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05" exitCode=0 Sep 10 19:35:15.419681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.419585 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" event={"ID":"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f","Type":"ContainerDied","Data":"0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05"} Sep 10 19:35:15.419681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.419597 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" Sep 10 19:35:15.419681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.419629 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t" event={"ID":"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f","Type":"ContainerDied","Data":"a9d0b3695bde8f22e48fc0050c7c87023fcfafd947ba7790e7a3987f304b3f2e"} Sep 10 19:35:15.419681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.419659 2584 scope.go:117] "RemoveContainer" containerID="0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05" Sep 10 19:35:15.428011 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.427991 2584 scope.go:117] "RemoveContainer" containerID="0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05" Sep 10 19:35:15.428330 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:35:15.428308 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05\": container with ID starting with 0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05 not found: ID does not exist" containerID="0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05" Sep 10 19:35:15.428392 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.428340 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05"} err="failed to get container status \"0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05\": rpc error: code = NotFound desc = could not find container \"0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05\": container with ID starting with 0ea12ee594350f1964d768dbb87903476bd8702729438cff74fa80f823951c05 not found: ID does not exist" Sep 10 19:35:15.561268 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.561238 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-plugins-config-volume\") pod \"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f\" (UID: \"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f\") " Sep 10 19:35:15.561473 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.561341 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t27p\" (UniqueName: \"kubernetes.io/projected/5d7dc494-c869-4776-99c6-125374169366-kube-api-access-5t27p\") pod \"5d7dc494-c869-4776-99c6-125374169366\" (UID: \"5d7dc494-c869-4776-99c6-125374169366\") " Sep 10 19:35:15.561473 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.561379 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z7ps\" (UniqueName: \"kubernetes.io/projected/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-kube-api-access-7z7ps\") pod \"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f\" (UID: \"f8b3408d-4991-4d9d-8f5a-833e05ee8c0f\") " Sep 10 19:35:15.561473 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.561403 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7dc494-c869-4776-99c6-125374169366-plugins-config-volume\") pod \"5d7dc494-c869-4776-99c6-125374169366\" (UID: \"5d7dc494-c869-4776-99c6-125374169366\") " Sep 10 19:35:15.561720 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.561685 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8b3408d-4991-4d9d-8f5a-833e05ee8c0f" (UID: "f8b3408d-4991-4d9d-8f5a-833e05ee8c0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:35:15.562057 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.562029 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7dc494-c869-4776-99c6-125374169366-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d7dc494-c869-4776-99c6-125374169366" (UID: "5d7dc494-c869-4776-99c6-125374169366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:35:15.564900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.564872 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7dc494-c869-4776-99c6-125374169366-kube-api-access-5t27p" (OuterVolumeSpecName: "kube-api-access-5t27p") pod "5d7dc494-c869-4776-99c6-125374169366" (UID: "5d7dc494-c869-4776-99c6-125374169366"). InnerVolumeSpecName "kube-api-access-5t27p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:35:15.565024 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.564979 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-kube-api-access-7z7ps" (OuterVolumeSpecName: "kube-api-access-7z7ps") pod "f8b3408d-4991-4d9d-8f5a-833e05ee8c0f" (UID: "f8b3408d-4991-4d9d-8f5a-833e05ee8c0f"). InnerVolumeSpecName "kube-api-access-7z7ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:35:15.663069 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.662961 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:35:15.663069 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.662998 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t27p\" (UniqueName: \"kubernetes.io/projected/5d7dc494-c869-4776-99c6-125374169366-kube-api-access-5t27p\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:35:15.663069 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.663011 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7ps\" (UniqueName: \"kubernetes.io/projected/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f-kube-api-access-7z7ps\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:35:15.663069 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.663025 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7dc494-c869-4776-99c6-125374169366-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:35:15.741127 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.741085 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t"] Sep 10 19:35:15.748176 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:15.748109 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7189cc9-568c9f66cf-stl5t"] Sep 10 19:35:16.425674 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:16.425643 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn" Sep 10 19:35:16.447442 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:16.447408 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn"] Sep 10 19:35:16.451322 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:16.451296 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7189cc9-9489c9bb-p9mgn"] Sep 10 19:35:16.488902 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:16.488861 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7dc494-c869-4776-99c6-125374169366" path="/var/lib/kubelet/pods/5d7dc494-c869-4776-99c6-125374169366/volumes" Sep 10 19:35:16.489388 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:16.489366 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b3408d-4991-4d9d-8f5a-833e05ee8c0f" path="/var/lib/kubelet/pods/f8b3408d-4991-4d9d-8f5a-833e05ee8c0f/volumes" Sep 10 19:35:32.682053 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.682012 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz"] Sep 10 19:35:32.682547 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.682408 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b3408d-4991-4d9d-8f5a-833e05ee8c0f" containerName="payload-processing" Sep 10 19:35:32.682547 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.682420 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b3408d-4991-4d9d-8f5a-833e05ee8c0f" containerName="payload-processing" Sep 10 19:35:32.682547 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.682438 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7dc494-c869-4776-99c6-125374169366" containerName="payload-pre-processing" Sep 10 19:35:32.682547 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.682444 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7dc494-c869-4776-99c6-125374169366" containerName="payload-pre-processing" Sep 10 19:35:32.682547 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.682500 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7dc494-c869-4776-99c6-125374169366" containerName="payload-pre-processing" Sep 10 19:35:32.682547 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.682511 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b3408d-4991-4d9d-8f5a-833e05ee8c0f" containerName="payload-processing" Sep 10 19:35:32.687134 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.687114 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.690818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.690792 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-18c3c3\"/\"openshift-service-ca.crt\"" Sep 10 19:35:32.691673 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.691643 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-18c3c3\"/\"default-dockercfg-wgswb\"" Sep 10 19:35:32.691804 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.691688 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-18c3c3\"/\"autores-model-18c3c3-kserve-self-signed-certs\"" Sep 10 19:35:32.691804 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.691721 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-18c3c3\"/\"kube-root-ca.crt\"" Sep 10 19:35:32.697593 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.697569 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz"] Sep 10 19:35:32.828024 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.827983 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zdqm\" (UniqueName: \"kubernetes.io/projected/c6b326d6-b8d5-44b0-871d-086b32bafff5-kube-api-access-9zdqm\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.828200 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.828045 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b326d6-b8d5-44b0-871d-086b32bafff5-tls-certs\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.828200 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.828081 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-tmp-dir\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.828200 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.828131 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-home\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.828200 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.828183 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-model-cache\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.828375 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.828219 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-dshm\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.929597 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.929558 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zdqm\" (UniqueName: \"kubernetes.io/projected/c6b326d6-b8d5-44b0-871d-086b32bafff5-kube-api-access-9zdqm\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.929772 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.929611 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b326d6-b8d5-44b0-871d-086b32bafff5-tls-certs\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.929772 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.929637 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-tmp-dir\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.929772 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.929681 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-home\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.929772 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.929702 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-model-cache\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.929772 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.929740 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-dshm\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.930123 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.930090 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-tmp-dir\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.930123 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.930113 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-model-cache\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.930273 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.930183 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-home\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.931907 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.931881 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-dshm\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.932351 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.932271 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b326d6-b8d5-44b0-871d-086b32bafff5-tls-certs\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.950973 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.950936 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zdqm\" (UniqueName: \"kubernetes.io/projected/c6b326d6-b8d5-44b0-871d-086b32bafff5-kube-api-access-9zdqm\") pod \"autores-model-18c3c3-kserve-7bd548d68c-jzhpz\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:32.996427 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:32.996388 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:33.286639 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:33.286605 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz"] Sep 10 19:35:33.286915 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:35:33.286888 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6b326d6_b8d5_44b0_871d_086b32bafff5.slice/crio-e848bb855c7125fa67282c3f4e8089bd895f98d1d224f860a4989dfb7b50fc1b WatchSource:0}: Error finding container e848bb855c7125fa67282c3f4e8089bd895f98d1d224f860a4989dfb7b50fc1b: Status 404 returned error can't find the container with id e848bb855c7125fa67282c3f4e8089bd895f98d1d224f860a4989dfb7b50fc1b Sep 10 19:35:33.511144 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:33.511104 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" event={"ID":"c6b326d6-b8d5-44b0-871d-086b32bafff5","Type":"ContainerStarted","Data":"f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c"} Sep 10 19:35:33.511376 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:33.511153 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" event={"ID":"c6b326d6-b8d5-44b0-871d-086b32bafff5","Type":"ContainerStarted","Data":"e848bb855c7125fa67282c3f4e8089bd895f98d1d224f860a4989dfb7b50fc1b"} Sep 10 19:35:33.533815 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:33.533713 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" podStartSLOduration=1.533694423 podStartE2EDuration="1.533694423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:33.533439263 +0000 UTC m=+919.501251509" watchObservedRunningTime="2026-09-10 19:35:33.533694423 +0000 UTC m=+919.501506675" Sep 10 19:35:42.996533 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:42.996485 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:42.996533 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:42.996543 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:43.009302 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:43.009250 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:43.573480 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:43.573443 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:35:44.959359 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:44.959325 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k"] Sep 10 19:35:44.962730 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:44.962705 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:44.965196 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:44.965168 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-f4a65e71-kserve-self-signed-certs\"" Sep 10 19:35:44.973005 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:44.972977 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k"] Sep 10 19:35:45.028484 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.028439 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-model-cache\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.028663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.028534 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-home\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.028663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.028577 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-dshm\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.028663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.028602 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-tmp-dir\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.028663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.028626 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bq9\" (UniqueName: \"kubernetes.io/projected/2f103535-3bf5-4fbd-b578-cffb71d313df-kube-api-access-m7bq9\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.028663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.028657 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f103535-3bf5-4fbd-b578-cffb71d313df-tls-certs\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.129524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.129483 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-home\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.129708 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.129534 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-dshm\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.129708 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.129561 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-tmp-dir\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.129708 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.129581 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bq9\" (UniqueName: \"kubernetes.io/projected/2f103535-3bf5-4fbd-b578-cffb71d313df-kube-api-access-m7bq9\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.129708 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.129605 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f103535-3bf5-4fbd-b578-cffb71d313df-tls-certs\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.129708 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.129649 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-model-cache\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.129964 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.129938 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-home\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.130010 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.129981 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-tmp-dir\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.130010 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.129992 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-model-cache\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.132042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.132009 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-dshm\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.132230 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.132216 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f103535-3bf5-4fbd-b578-cffb71d313df-tls-certs\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.138344 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.138317 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bq9\" (UniqueName: \"kubernetes.io/projected/2f103535-3bf5-4fbd-b578-cffb71d313df-kube-api-access-m7bq9\") pod \"e2e-conflict-b-f4a65e71-kserve-5766966686-v474k\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.273951 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.273849 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:45.548890 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.548858 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k"] Sep 10 19:35:45.551526 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:35:45.551489 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f103535_3bf5_4fbd_b578_cffb71d313df.slice/crio-03e2aa348cc83b141f8dbbe9028d0721cca8bb257d43f16320bb2835eb41971d WatchSource:0}: Error finding container 03e2aa348cc83b141f8dbbe9028d0721cca8bb257d43f16320bb2835eb41971d: Status 404 returned error can't find the container with id 03e2aa348cc83b141f8dbbe9028d0721cca8bb257d43f16320bb2835eb41971d Sep 10 19:35:45.574098 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:45.574066 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" event={"ID":"2f103535-3bf5-4fbd-b578-cffb71d313df","Type":"ContainerStarted","Data":"03e2aa348cc83b141f8dbbe9028d0721cca8bb257d43f16320bb2835eb41971d"} Sep 10 19:35:46.580379 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:46.580333 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" event={"ID":"2f103535-3bf5-4fbd-b578-cffb71d313df","Type":"ContainerStarted","Data":"0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd"} Sep 10 19:35:46.596689 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:46.596623 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" podStartSLOduration=2.596605686 podStartE2EDuration="2.596605686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:46.595477632 +0000 UTC m=+932.563289873" watchObservedRunningTime="2026-09-10 19:35:46.596605686 +0000 UTC m=+932.564417996" Sep 10 19:35:55.274424 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:55.274375 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:55.274424 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:55.274435 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:55.287216 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:55.287184 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:55.636214 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:55.636173 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:35:57.016576 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.016532 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j"] Sep 10 19:35:57.020195 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.020170 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:35:57.022874 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.022851 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b06cf0bb\"" Sep 10 19:35:57.023003 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.022985 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b06cf0bb-dockercfg-lwhxc\"" Sep 10 19:35:57.032470 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.032435 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j"] Sep 10 19:35:57.145802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.145764 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1d0c81-47b2-4c20-ba85-fc504634c848-plugins-config-volume\") pod \"payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j\" (UID: \"1f1d0c81-47b2-4c20-ba85-fc504634c848\") " pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:35:57.145986 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.145882 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w99f\" (UniqueName: \"kubernetes.io/projected/1f1d0c81-47b2-4c20-ba85-fc504634c848-kube-api-access-9w99f\") pod \"payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j\" (UID: \"1f1d0c81-47b2-4c20-ba85-fc504634c848\") " pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:35:57.247366 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.247331 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w99f\" (UniqueName: \"kubernetes.io/projected/1f1d0c81-47b2-4c20-ba85-fc504634c848-kube-api-access-9w99f\") pod \"payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j\" (UID: \"1f1d0c81-47b2-4c20-ba85-fc504634c848\") " pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:35:57.247580 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.247460 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1d0c81-47b2-4c20-ba85-fc504634c848-plugins-config-volume\") pod \"payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j\" (UID: \"1f1d0c81-47b2-4c20-ba85-fc504634c848\") " pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:35:57.248230 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.248202 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1d0c81-47b2-4c20-ba85-fc504634c848-plugins-config-volume\") pod \"payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j\" (UID: \"1f1d0c81-47b2-4c20-ba85-fc504634c848\") " pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:35:57.256109 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.256083 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w99f\" (UniqueName: \"kubernetes.io/projected/1f1d0c81-47b2-4c20-ba85-fc504634c848-kube-api-access-9w99f\") pod \"payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j\" (UID: \"1f1d0c81-47b2-4c20-ba85-fc504634c848\") " pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:35:57.332813 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.332771 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:35:57.607027 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.606941 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j"] Sep 10 19:35:57.610586 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:35:57.610554 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1d0c81_47b2_4c20_ba85_fc504634c848.slice/crio-48e62a47d75d58a1e7357637ffff1246fd93710b54e16114dfd8e8ed9da57df0 WatchSource:0}: Error finding container 48e62a47d75d58a1e7357637ffff1246fd93710b54e16114dfd8e8ed9da57df0: Status 404 returned error can't find the container with id 48e62a47d75d58a1e7357637ffff1246fd93710b54e16114dfd8e8ed9da57df0 Sep 10 19:35:57.636560 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:57.636521 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" event={"ID":"1f1d0c81-47b2-4c20-ba85-fc504634c848","Type":"ContainerStarted","Data":"48e62a47d75d58a1e7357637ffff1246fd93710b54e16114dfd8e8ed9da57df0"} Sep 10 19:35:58.642990 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:58.642950 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" event={"ID":"1f1d0c81-47b2-4c20-ba85-fc504634c848","Type":"ContainerStarted","Data":"3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e"} Sep 10 19:35:58.643522 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:35:58.643010 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:36:00.951682 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:00.951599 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" podStartSLOduration=4.951574084 podStartE2EDuration="4.951574084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:58.666151776 +0000 UTC m=+944.633964014" watchObservedRunningTime="2026-09-10 19:36:00.951574084 +0000 UTC m=+946.919386332" Sep 10 19:36:00.955452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:00.955414 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz"] Sep 10 19:36:04.770680 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:04.770636 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k"] Sep 10 19:36:09.650530 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:09.650496 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:36:11.723069 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:11.723033 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn"] Sep 10 19:36:11.723572 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:11.723263 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" podUID="941884eb-043a-411b-88eb-ebc67ec1f7fc" containerName="payload-pre-processing" containerID="cri-o://3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3" gracePeriod=30 Sep 10 19:36:11.965520 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:11.965490 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:36:12.091849 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.091799 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dnln\" (UniqueName: \"kubernetes.io/projected/941884eb-043a-411b-88eb-ebc67ec1f7fc-kube-api-access-5dnln\") pod \"941884eb-043a-411b-88eb-ebc67ec1f7fc\" (UID: \"941884eb-043a-411b-88eb-ebc67ec1f7fc\") " Sep 10 19:36:12.092030 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.091883 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941884eb-043a-411b-88eb-ebc67ec1f7fc-plugins-config-volume\") pod \"941884eb-043a-411b-88eb-ebc67ec1f7fc\" (UID: \"941884eb-043a-411b-88eb-ebc67ec1f7fc\") " Sep 10 19:36:12.092309 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.092259 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941884eb-043a-411b-88eb-ebc67ec1f7fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "941884eb-043a-411b-88eb-ebc67ec1f7fc" (UID: "941884eb-043a-411b-88eb-ebc67ec1f7fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:36:12.094013 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.093985 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941884eb-043a-411b-88eb-ebc67ec1f7fc-kube-api-access-5dnln" (OuterVolumeSpecName: "kube-api-access-5dnln") pod "941884eb-043a-411b-88eb-ebc67ec1f7fc" (UID: "941884eb-043a-411b-88eb-ebc67ec1f7fc"). InnerVolumeSpecName "kube-api-access-5dnln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:36:12.192853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.192800 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dnln\" (UniqueName: \"kubernetes.io/projected/941884eb-043a-411b-88eb-ebc67ec1f7fc-kube-api-access-5dnln\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:12.192853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.192843 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941884eb-043a-411b-88eb-ebc67ec1f7fc-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:12.725037 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.725004 2584 generic.go:356] "Generic (PLEG): container finished" podID="941884eb-043a-411b-88eb-ebc67ec1f7fc" containerID="3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3" exitCode=0 Sep 10 19:36:12.725502 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.725056 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" Sep 10 19:36:12.725502 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.725085 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" event={"ID":"941884eb-043a-411b-88eb-ebc67ec1f7fc","Type":"ContainerDied","Data":"3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3"} Sep 10 19:36:12.725502 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.725140 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn" event={"ID":"941884eb-043a-411b-88eb-ebc67ec1f7fc","Type":"ContainerDied","Data":"dfb1c84f66a7c11a33c4640f98af9f31173617454c8d0235bedc057ca6339ded"} Sep 10 19:36:12.725502 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.725163 2584 scope.go:117] "RemoveContainer" containerID="3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3" Sep 10 19:36:12.734021 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.734001 2584 scope.go:117] "RemoveContainer" containerID="3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3" Sep 10 19:36:12.734337 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:36:12.734316 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3\": container with ID starting with 3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3 not found: ID does not exist" containerID="3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3" Sep 10 19:36:12.734394 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.734348 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3"} err="failed to get container status \"3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3\": rpc error: code = NotFound desc = could not find container \"3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3\": container with ID starting with 3c9fe96ce2a49db3ba98b2179f25396242f0d6cc48d88c6da0a25b3b776946e3 not found: ID does not exist" Sep 10 19:36:12.741735 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.741702 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn"] Sep 10 19:36:12.746062 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:12.745991 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-18c3c3-846c774f46-vqgtn"] Sep 10 19:36:14.490007 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:14.489970 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="941884eb-043a-411b-88eb-ebc67ec1f7fc" path="/var/lib/kubelet/pods/941884eb-043a-411b-88eb-ebc67ec1f7fc/volumes" Sep 10 19:36:15.976158 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:15.976112 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" podUID="c6b326d6-b8d5-44b0-871d-086b32bafff5" containerName="main" containerID="cri-o://f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c" gracePeriod=45 Sep 10 19:36:16.239671 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.239604 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:36:16.331417 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.331362 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zdqm\" (UniqueName: \"kubernetes.io/projected/c6b326d6-b8d5-44b0-871d-086b32bafff5-kube-api-access-9zdqm\") pod \"c6b326d6-b8d5-44b0-871d-086b32bafff5\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " Sep 10 19:36:16.331417 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.331424 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-home\") pod \"c6b326d6-b8d5-44b0-871d-086b32bafff5\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " Sep 10 19:36:16.331662 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.331527 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b326d6-b8d5-44b0-871d-086b32bafff5-tls-certs\") pod \"c6b326d6-b8d5-44b0-871d-086b32bafff5\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " Sep 10 19:36:16.331662 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.331542 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-tmp-dir\") pod \"c6b326d6-b8d5-44b0-871d-086b32bafff5\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " Sep 10 19:36:16.331662 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.331563 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-dshm\") pod \"c6b326d6-b8d5-44b0-871d-086b32bafff5\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " Sep 10 19:36:16.331662 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.331619 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-model-cache\") pod \"c6b326d6-b8d5-44b0-871d-086b32bafff5\" (UID: \"c6b326d6-b8d5-44b0-871d-086b32bafff5\") " Sep 10 19:36:16.331866 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.331710 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-home" (OuterVolumeSpecName: "home") pod "c6b326d6-b8d5-44b0-871d-086b32bafff5" (UID: "c6b326d6-b8d5-44b0-871d-086b32bafff5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:36:16.331866 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.331854 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6b326d6-b8d5-44b0-871d-086b32bafff5" (UID: "c6b326d6-b8d5-44b0-871d-086b32bafff5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:36:16.331988 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.331939 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6b326d6-b8d5-44b0-871d-086b32bafff5" (UID: "c6b326d6-b8d5-44b0-871d-086b32bafff5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:36:16.332044 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.332001 2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-tmp-dir\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:16.332044 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.332020 2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-home\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:16.333866 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.333842 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b326d6-b8d5-44b0-871d-086b32bafff5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6b326d6-b8d5-44b0-871d-086b32bafff5" (UID: "c6b326d6-b8d5-44b0-871d-086b32bafff5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:36:16.333866 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.333850 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b326d6-b8d5-44b0-871d-086b32bafff5-kube-api-access-9zdqm" (OuterVolumeSpecName: "kube-api-access-9zdqm") pod "c6b326d6-b8d5-44b0-871d-086b32bafff5" (UID: "c6b326d6-b8d5-44b0-871d-086b32bafff5"). InnerVolumeSpecName "kube-api-access-9zdqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:36:16.333866 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.333852 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-dshm" (OuterVolumeSpecName: "dshm") pod "c6b326d6-b8d5-44b0-871d-086b32bafff5" (UID: "c6b326d6-b8d5-44b0-871d-086b32bafff5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:36:16.432855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.432812 2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-model-cache\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:16.432855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.432846 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zdqm\" (UniqueName: \"kubernetes.io/projected/c6b326d6-b8d5-44b0-871d-086b32bafff5-kube-api-access-9zdqm\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:16.432855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.432856 2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b326d6-b8d5-44b0-871d-086b32bafff5-tls-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:16.432855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.432865 2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6b326d6-b8d5-44b0-871d-086b32bafff5-dshm\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:16.747108 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.747018 2584 generic.go:356] "Generic (PLEG): container finished" podID="c6b326d6-b8d5-44b0-871d-086b32bafff5" containerID="f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c" exitCode=0 Sep 10 19:36:16.747108 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.747070 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" Sep 10 19:36:16.747327 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.747104 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" event={"ID":"c6b326d6-b8d5-44b0-871d-086b32bafff5","Type":"ContainerDied","Data":"f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c"} Sep 10 19:36:16.747327 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.747151 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz" event={"ID":"c6b326d6-b8d5-44b0-871d-086b32bafff5","Type":"ContainerDied","Data":"e848bb855c7125fa67282c3f4e8089bd895f98d1d224f860a4989dfb7b50fc1b"} Sep 10 19:36:16.747327 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.747177 2584 scope.go:117] "RemoveContainer" containerID="f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c" Sep 10 19:36:16.756016 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.755996 2584 scope.go:117] "RemoveContainer" containerID="f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c" Sep 10 19:36:16.756405 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:36:16.756377 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c\": container with ID starting with f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c not found: ID does not exist" containerID="f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c" Sep 10 19:36:16.756495 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.756415 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c"} err="failed to get container status \"f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c\": rpc error: code = NotFound desc = could not find container \"f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c\": container with ID starting with f3406cc565598d851632be2951b028c8e023a628384f2cfe28d730de7eb6a11c not found: ID does not exist" Sep 10 19:36:16.762259 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.762224 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz"] Sep 10 19:36:16.768156 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:16.768128 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-18c3c3/autores-model-18c3c3-kserve-7bd548d68c-jzhpz"] Sep 10 19:36:18.490677 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:18.490640 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6b326d6-b8d5-44b0-871d-086b32bafff5" path="/var/lib/kubelet/pods/c6b326d6-b8d5-44b0-871d-086b32bafff5/volumes" Sep 10 19:36:19.790421 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:19.790382 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" podUID="2f103535-3bf5-4fbd-b578-cffb71d313df" containerName="main" containerID="cri-o://0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd" gracePeriod=45 Sep 10 19:36:20.040555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.040493 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:36:20.169659 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.169617 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7bq9\" (UniqueName: \"kubernetes.io/projected/2f103535-3bf5-4fbd-b578-cffb71d313df-kube-api-access-m7bq9\") pod \"2f103535-3bf5-4fbd-b578-cffb71d313df\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " Sep 10 19:36:20.169659 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.169664 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-tmp-dir\") pod \"2f103535-3bf5-4fbd-b578-cffb71d313df\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " Sep 10 19:36:20.169867 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.169694 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-dshm\") pod \"2f103535-3bf5-4fbd-b578-cffb71d313df\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " Sep 10 19:36:20.169867 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.169824 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-model-cache\") pod \"2f103535-3bf5-4fbd-b578-cffb71d313df\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " Sep 10 19:36:20.169867 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.169859 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-home\") pod \"2f103535-3bf5-4fbd-b578-cffb71d313df\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " Sep 10 19:36:20.170015 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.169885 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f103535-3bf5-4fbd-b578-cffb71d313df-tls-certs\") pod \"2f103535-3bf5-4fbd-b578-cffb71d313df\" (UID: \"2f103535-3bf5-4fbd-b578-cffb71d313df\") " Sep 10 19:36:20.170015 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.169934 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f103535-3bf5-4fbd-b578-cffb71d313df" (UID: "2f103535-3bf5-4fbd-b578-cffb71d313df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:36:20.170119 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.170084 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f103535-3bf5-4fbd-b578-cffb71d313df" (UID: "2f103535-3bf5-4fbd-b578-cffb71d313df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:36:20.170208 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.170192 2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-model-cache\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:20.170272 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.170209 2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-tmp-dir\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:20.170409 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.170388 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-home" (OuterVolumeSpecName: "home") pod "2f103535-3bf5-4fbd-b578-cffb71d313df" (UID: "2f103535-3bf5-4fbd-b578-cffb71d313df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:36:20.172008 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.171975 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-dshm" (OuterVolumeSpecName: "dshm") pod "2f103535-3bf5-4fbd-b578-cffb71d313df" (UID: "2f103535-3bf5-4fbd-b578-cffb71d313df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:36:20.172097 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.172019 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f103535-3bf5-4fbd-b578-cffb71d313df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f103535-3bf5-4fbd-b578-cffb71d313df" (UID: "2f103535-3bf5-4fbd-b578-cffb71d313df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:36:20.172097 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.172065 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f103535-3bf5-4fbd-b578-cffb71d313df-kube-api-access-m7bq9" (OuterVolumeSpecName: "kube-api-access-m7bq9") pod "2f103535-3bf5-4fbd-b578-cffb71d313df" (UID: "2f103535-3bf5-4fbd-b578-cffb71d313df"). InnerVolumeSpecName "kube-api-access-m7bq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:36:20.271739 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.271698 2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-home\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:20.271739 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.271737 2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f103535-3bf5-4fbd-b578-cffb71d313df-tls-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:20.271739 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.271751 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7bq9\" (UniqueName: \"kubernetes.io/projected/2f103535-3bf5-4fbd-b578-cffb71d313df-kube-api-access-m7bq9\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:20.271979 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.271760 2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f103535-3bf5-4fbd-b578-cffb71d313df-dshm\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:20.770665 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.770568 2584 generic.go:356] "Generic (PLEG): container finished" podID="2f103535-3bf5-4fbd-b578-cffb71d313df" containerID="0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd" exitCode=0 Sep 10 19:36:20.770665 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.770611 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" Sep 10 19:36:20.770665 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.770643 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" event={"ID":"2f103535-3bf5-4fbd-b578-cffb71d313df","Type":"ContainerDied","Data":"0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd"} Sep 10 19:36:20.770952 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.770687 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k" event={"ID":"2f103535-3bf5-4fbd-b578-cffb71d313df","Type":"ContainerDied","Data":"03e2aa348cc83b141f8dbbe9028d0721cca8bb257d43f16320bb2835eb41971d"} Sep 10 19:36:20.770952 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.770704 2584 scope.go:117] "RemoveContainer" containerID="0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd" Sep 10 19:36:20.779592 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.779570 2584 scope.go:117] "RemoveContainer" containerID="0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd" Sep 10 19:36:20.779918 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:36:20.779895 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd\": container with ID starting with 0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd not found: ID does not exist" containerID="0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd" Sep 10 19:36:20.779965 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.779928 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd"} err="failed to get container status \"0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd\": rpc error: code = NotFound desc = could not find container \"0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd\": container with ID starting with 0e2be1c90ecbe36918bd51dbbdb8b1b1e78131054b6444f28998d230116ee8bd not found: ID does not exist" Sep 10 19:36:20.785965 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.785939 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k"] Sep 10 19:36:20.791901 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:20.791874 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-f4a65e71-kserve-5766966686-v474k"] Sep 10 19:36:22.488623 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:22.488583 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f103535-3bf5-4fbd-b578-cffb71d313df" path="/var/lib/kubelet/pods/2f103535-3bf5-4fbd-b578-cffb71d313df/volumes" Sep 10 19:36:27.035514 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.035475 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j"] Sep 10 19:36:27.036108 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.035707 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" podUID="1f1d0c81-47b2-4c20-ba85-fc504634c848" containerName="payload-processing" containerID="cri-o://3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e" gracePeriod=30 Sep 10 19:36:27.282257 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.282231 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:36:27.332822 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.332783 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1d0c81-47b2-4c20-ba85-fc504634c848-plugins-config-volume\") pod \"1f1d0c81-47b2-4c20-ba85-fc504634c848\" (UID: \"1f1d0c81-47b2-4c20-ba85-fc504634c848\") " Sep 10 19:36:27.333034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.332887 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w99f\" (UniqueName: \"kubernetes.io/projected/1f1d0c81-47b2-4c20-ba85-fc504634c848-kube-api-access-9w99f\") pod \"1f1d0c81-47b2-4c20-ba85-fc504634c848\" (UID: \"1f1d0c81-47b2-4c20-ba85-fc504634c848\") " Sep 10 19:36:27.333216 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.333188 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1d0c81-47b2-4c20-ba85-fc504634c848-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f1d0c81-47b2-4c20-ba85-fc504634c848" (UID: "1f1d0c81-47b2-4c20-ba85-fc504634c848"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:36:27.335018 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.334991 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1d0c81-47b2-4c20-ba85-fc504634c848-kube-api-access-9w99f" (OuterVolumeSpecName: "kube-api-access-9w99f") pod "1f1d0c81-47b2-4c20-ba85-fc504634c848" (UID: "1f1d0c81-47b2-4c20-ba85-fc504634c848"). InnerVolumeSpecName "kube-api-access-9w99f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:36:27.434457 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.434419 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w99f\" (UniqueName: \"kubernetes.io/projected/1f1d0c81-47b2-4c20-ba85-fc504634c848-kube-api-access-9w99f\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:27.434457 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.434448 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1d0c81-47b2-4c20-ba85-fc504634c848-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:36:27.814634 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.814599 2584 generic.go:356] "Generic (PLEG): container finished" podID="1f1d0c81-47b2-4c20-ba85-fc504634c848" containerID="3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e" exitCode=0 Sep 10 19:36:27.814855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.814647 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" event={"ID":"1f1d0c81-47b2-4c20-ba85-fc504634c848","Type":"ContainerDied","Data":"3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e"} Sep 10 19:36:27.814855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.814672 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" Sep 10 19:36:27.814855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.814689 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j" event={"ID":"1f1d0c81-47b2-4c20-ba85-fc504634c848","Type":"ContainerDied","Data":"48e62a47d75d58a1e7357637ffff1246fd93710b54e16114dfd8e8ed9da57df0"} Sep 10 19:36:27.814855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.814710 2584 scope.go:117] "RemoveContainer" containerID="3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e" Sep 10 19:36:27.823646 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.823620 2584 scope.go:117] "RemoveContainer" containerID="3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e" Sep 10 19:36:27.824077 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:36:27.823942 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e\": container with ID starting with 3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e not found: ID does not exist" containerID="3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e" Sep 10 19:36:27.824077 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.823975 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e"} err="failed to get container status \"3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e\": rpc error: code = NotFound desc = could not find container \"3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e\": container with ID starting with 3b6373ce09cd3585cf627a5936969de7f78b287d66574552a1557a932fec880e not found: ID does not exist" Sep 10 19:36:27.838450 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.838419 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j"] Sep 10 19:36:27.841950 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:27.841922 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b06cf0bb-6849bb46bf-zv22j"] Sep 10 19:36:28.488781 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:28.488745 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f1d0c81-47b2-4c20-ba85-fc504634c848" path="/var/lib/kubelet/pods/1f1d0c81-47b2-4c20-ba85-fc504634c848/volumes" Sep 10 19:36:29.128955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.128922 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr"] Sep 10 19:36:29.129345 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129331 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f103535-3bf5-4fbd-b578-cffb71d313df" containerName="main" Sep 10 19:36:29.129411 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129347 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f103535-3bf5-4fbd-b578-cffb71d313df" containerName="main" Sep 10 19:36:29.129411 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129359 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f1d0c81-47b2-4c20-ba85-fc504634c848" containerName="payload-processing" Sep 10 19:36:29.129411 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129364 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f1d0c81-47b2-4c20-ba85-fc504634c848" containerName="payload-processing" Sep 10 19:36:29.129411 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129383 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b326d6-b8d5-44b0-871d-086b32bafff5" containerName="main" Sep 10 19:36:29.129411 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129389 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b326d6-b8d5-44b0-871d-086b32bafff5" containerName="main" Sep 10 19:36:29.129411 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129397 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941884eb-043a-411b-88eb-ebc67ec1f7fc" containerName="payload-pre-processing" Sep 10 19:36:29.129411 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129403 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="941884eb-043a-411b-88eb-ebc67ec1f7fc" containerName="payload-pre-processing" Sep 10 19:36:29.129635 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129473 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b326d6-b8d5-44b0-871d-086b32bafff5" containerName="main" Sep 10 19:36:29.129635 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129483 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f103535-3bf5-4fbd-b578-cffb71d313df" containerName="main" Sep 10 19:36:29.129635 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129490 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="941884eb-043a-411b-88eb-ebc67ec1f7fc" containerName="payload-pre-processing" Sep 10 19:36:29.129635 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.129496 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f1d0c81-47b2-4c20-ba85-fc504634c848" containerName="payload-processing" Sep 10 19:36:29.134179 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.134158 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.137360 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.137333 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-cdcbd691-openshift-default-dockercfg-tbr89\"" Sep 10 19:36:29.149342 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.149309 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-podinfo\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.149476 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.149352 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-certs\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.149476 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.149373 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhzz\" (UniqueName: \"kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-kube-api-access-xkhzz\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.149476 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.149436 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-socket\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.149476 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.149463 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-credential-socket\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.149656 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.149483 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istiod-ca-cert\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.149656 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.149519 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-token\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.149656 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.149595 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-data\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.149656 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.149632 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-envoy\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.154368 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.154325 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr"] Sep 10 19:36:29.250538 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.250502 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-podinfo\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.250742 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.250553 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-certs\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.250742 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.250681 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhzz\" (UniqueName: \"kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-kube-api-access-xkhzz\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.250877 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.250778 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-socket\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.250877 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.250808 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-credential-socket\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.250877 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.250839 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istiod-ca-cert\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.251023 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.250887 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-certs\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.251023 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.250894 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-token\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.251023 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.250958 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-data\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.251023 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.251020 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-envoy\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.251218 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.251137 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-socket\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.251274 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.251255 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-credential-socket\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.251387 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.251363 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-data\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.251621 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.251597 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istiod-ca-cert\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.253225 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.253197 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-envoy\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.253367 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.253348 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-podinfo\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.262827 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.262804 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-token\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.263452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.263431 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhzz\" (UniqueName: \"kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-kube-api-access-xkhzz\") pod \"e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.445331 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.445215 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:29.730107 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.730012 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr"] Sep 10 19:36:29.733511 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:36:29.733482 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a276e15_eaf4_4d86_9600_9ee3c353f2ef.slice/crio-a2df0915939fad0e3fb97602824a80e4eb5e4ce26023ba5aecfb75827b6b0da4 WatchSource:0}: Error finding container a2df0915939fad0e3fb97602824a80e4eb5e4ce26023ba5aecfb75827b6b0da4: Status 404 returned error can't find the container with id a2df0915939fad0e3fb97602824a80e4eb5e4ce26023ba5aecfb75827b6b0da4 Sep 10 19:36:29.735650 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.735612 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:36:29.735724 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.735690 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:36:29.735724 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.735720 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:36:29.829159 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.829118 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" event={"ID":"7a276e15-eaf4-4d86-9600-9ee3c353f2ef","Type":"ContainerStarted","Data":"0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c"} Sep 10 19:36:29.829336 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.829177 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" event={"ID":"7a276e15-eaf4-4d86-9600-9ee3c353f2ef","Type":"ContainerStarted","Data":"a2df0915939fad0e3fb97602824a80e4eb5e4ce26023ba5aecfb75827b6b0da4"} Sep 10 19:36:29.849743 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:29.849679 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" podStartSLOduration=0.849662825 podStartE2EDuration="849.662825ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:36:29.847851618 +0000 UTC m=+975.815663866" watchObservedRunningTime="2026-09-10 19:36:29.849662825 +0000 UTC m=+975.817475072" Sep 10 19:36:30.446083 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:30.446044 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:30.451274 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:30.451246 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:30.834648 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:30.834618 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:30.835758 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:30.835737 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:33.052960 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.052922 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht"] Sep 10 19:36:33.056556 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.056537 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:36:33.058804 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.058779 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-cdcbd691\"" Sep 10 19:36:33.058933 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.058916 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-cdcbd691-dockercfg-n8hf4\"" Sep 10 19:36:33.065381 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.065353 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht"] Sep 10 19:36:33.087660 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.087625 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21eca854-5ee5-499d-9dd4-67d5a1aba957-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht\" (UID: \"21eca854-5ee5-499d-9dd4-67d5a1aba957\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:36:33.087834 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.087671 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q8ms\" (UniqueName: \"kubernetes.io/projected/21eca854-5ee5-499d-9dd4-67d5a1aba957-kube-api-access-4q8ms\") pod \"payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht\" (UID: \"21eca854-5ee5-499d-9dd4-67d5a1aba957\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:36:33.188461 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.188422 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21eca854-5ee5-499d-9dd4-67d5a1aba957-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht\" (UID: \"21eca854-5ee5-499d-9dd4-67d5a1aba957\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:36:33.188639 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.188471 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8ms\" (UniqueName: \"kubernetes.io/projected/21eca854-5ee5-499d-9dd4-67d5a1aba957-kube-api-access-4q8ms\") pod \"payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht\" (UID: \"21eca854-5ee5-499d-9dd4-67d5a1aba957\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:36:33.189138 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.189117 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21eca854-5ee5-499d-9dd4-67d5a1aba957-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht\" (UID: \"21eca854-5ee5-499d-9dd4-67d5a1aba957\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:36:33.196929 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.196899 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8ms\" (UniqueName: \"kubernetes.io/projected/21eca854-5ee5-499d-9dd4-67d5a1aba957-kube-api-access-4q8ms\") pod \"payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht\" (UID: \"21eca854-5ee5-499d-9dd4-67d5a1aba957\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:36:33.369245 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.369146 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:36:33.643022 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.642932 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht"] Sep 10 19:36:33.646552 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:36:33.646519 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21eca854_5ee5_499d_9dd4_67d5a1aba957.slice/crio-39140e94bfed09d489d2f5789f8fabb14f3c0bc8161c7b357ad91ee26747d581 WatchSource:0}: Error finding container 39140e94bfed09d489d2f5789f8fabb14f3c0bc8161c7b357ad91ee26747d581: Status 404 returned error can't find the container with id 39140e94bfed09d489d2f5789f8fabb14f3c0bc8161c7b357ad91ee26747d581 Sep 10 19:36:33.852273 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.852228 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" event={"ID":"21eca854-5ee5-499d-9dd4-67d5a1aba957","Type":"ContainerStarted","Data":"6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227"} Sep 10 19:36:33.852457 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.852339 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" event={"ID":"21eca854-5ee5-499d-9dd4-67d5a1aba957","Type":"ContainerStarted","Data":"39140e94bfed09d489d2f5789f8fabb14f3c0bc8161c7b357ad91ee26747d581"} Sep 10 19:36:33.852457 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.852402 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:36:33.870271 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:33.870214 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" podStartSLOduration=0.870194108 podStartE2EDuration="870.194108ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:36:33.868919833 +0000 UTC m=+979.836732091" watchObservedRunningTime="2026-09-10 19:36:33.870194108 +0000 UTC m=+979.838006358" Sep 10 19:36:44.861779 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:44.861746 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:36:55.987200 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:55.987163 2584 generic.go:356] "Generic (PLEG): container finished" podID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerID="0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c" exitCode=0 Sep 10 19:36:55.987653 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:55.987235 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" event={"ID":"7a276e15-eaf4-4d86-9600-9ee3c353f2ef","Type":"ContainerDied","Data":"0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c"} Sep 10 19:36:55.987800 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:55.987768 2584 scope.go:117] "RemoveContainer" containerID="0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c" Sep 10 19:36:55.988965 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:55.988934 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:36:55.989044 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:55.989025 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:36:55.989081 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:55.989066 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:36:56.035183 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.035146 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx"] Sep 10 19:36:56.038766 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.038741 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" Sep 10 19:36:56.041086 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.041061 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:36:56.041322 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.041115 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zztzd\"" Sep 10 19:36:56.045025 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.044995 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx"] Sep 10 19:36:56.215129 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.215078 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6167b1d9-438e-4987-9650-d3690d97572a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx\" (UID: \"6167b1d9-438e-4987-9650-d3690d97572a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" Sep 10 19:36:56.315802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.315759 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6167b1d9-438e-4987-9650-d3690d97572a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx\" (UID: \"6167b1d9-438e-4987-9650-d3690d97572a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" Sep 10 19:36:56.316451 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.316429 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6167b1d9-438e-4987-9650-d3690d97572a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx\" (UID: \"6167b1d9-438e-4987-9650-d3690d97572a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" Sep 10 19:36:56.355798 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.355753 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" Sep 10 19:36:56.634849 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.634812 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx"] Sep 10 19:36:56.993563 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.993456 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" event={"ID":"6167b1d9-438e-4987-9650-d3690d97572a","Type":"ContainerStarted","Data":"b494cfb58bc72d9c8f99eb46ebb3281581e46a448923d69fa1c11019389bd793"} Sep 10 19:36:56.995482 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:56.995457 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" event={"ID":"7a276e15-eaf4-4d86-9600-9ee3c353f2ef","Type":"ContainerStarted","Data":"70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589"} Sep 10 19:36:57.445543 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:57.445423 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:36:57.446859 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:57.446826 2584 patch_prober.go:28] interesting pod/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 19:36:57.447009 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:57.446888 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 19:36:58.446778 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:58.446253 2584 patch_prober.go:28] interesting pod/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 19:36:58.446778 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:58.446343 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 19:36:59.446427 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:59.446387 2584 patch_prober.go:28] interesting pod/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 19:36:59.446602 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:36:59.446443 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 19:37:00.016053 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:00.015100 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" event={"ID":"6167b1d9-438e-4987-9650-d3690d97572a","Type":"ContainerStarted","Data":"eb144481452dcba9c79169e74defca91dd54624a265bd885fa51ecd91729baa2"} Sep 10 19:37:00.446174 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:00.446130 2584 patch_prober.go:28] interesting pod/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 19:37:00.446380 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:00.446196 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 19:37:00.834714 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:00.834659 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:37:01.025716 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:01.025672 2584 generic.go:356] "Generic (PLEG): container finished" podID="6167b1d9-438e-4987-9650-d3690d97572a" containerID="eb144481452dcba9c79169e74defca91dd54624a265bd885fa51ecd91729baa2" exitCode=0 Sep 10 19:37:01.026189 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:01.025780 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" event={"ID":"6167b1d9-438e-4987-9650-d3690d97572a","Type":"ContainerDied","Data":"eb144481452dcba9c79169e74defca91dd54624a265bd885fa51ecd91729baa2"} Sep 10 19:37:01.446101 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:01.446062 2584 patch_prober.go:28] interesting pod/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 19:37:01.446296 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:01.446126 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 19:37:02.187196 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:02.187166 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" Sep 10 19:37:02.283710 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:02.283659 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6167b1d9-438e-4987-9650-d3690d97572a-maas-api-service-ca\") pod \"6167b1d9-438e-4987-9650-d3690d97572a\" (UID: \"6167b1d9-438e-4987-9650-d3690d97572a\") " Sep 10 19:37:02.284046 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:02.284019 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6167b1d9-438e-4987-9650-d3690d97572a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6167b1d9-438e-4987-9650-d3690d97572a" (UID: "6167b1d9-438e-4987-9650-d3690d97572a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:37:02.385111 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:02.385063 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6167b1d9-438e-4987-9650-d3690d97572a-maas-api-service-ca\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:02.451346 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:02.451313 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:37:02.452367 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:02.452344 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:37:03.041266 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:03.041229 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" event={"ID":"6167b1d9-438e-4987-9650-d3690d97572a","Type":"ContainerDied","Data":"b494cfb58bc72d9c8f99eb46ebb3281581e46a448923d69fa1c11019389bd793"} Sep 10 19:37:03.041266 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:03.041251 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx" Sep 10 19:37:03.041266 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:03.041298 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b494cfb58bc72d9c8f99eb46ebb3281581e46a448923d69fa1c11019389bd793" Sep 10 19:37:13.003112 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.003076 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht"] Sep 10 19:37:13.003551 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.003338 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" podUID="21eca854-5ee5-499d-9dd4-67d5a1aba957" containerName="payload-pre-processing" containerID="cri-o://6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227" gracePeriod=30 Sep 10 19:37:13.174714 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.174677 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx"] Sep 10 19:37:13.177971 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.177938 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cdcbd691-rm6sx"] Sep 10 19:37:13.260105 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.260038 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:37:13.289915 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.289876 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21eca854-5ee5-499d-9dd4-67d5a1aba957-plugins-config-volume\") pod \"21eca854-5ee5-499d-9dd4-67d5a1aba957\" (UID: \"21eca854-5ee5-499d-9dd4-67d5a1aba957\") " Sep 10 19:37:13.290087 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.290002 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q8ms\" (UniqueName: \"kubernetes.io/projected/21eca854-5ee5-499d-9dd4-67d5a1aba957-kube-api-access-4q8ms\") pod \"21eca854-5ee5-499d-9dd4-67d5a1aba957\" (UID: \"21eca854-5ee5-499d-9dd4-67d5a1aba957\") " Sep 10 19:37:13.290328 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.290299 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21eca854-5ee5-499d-9dd4-67d5a1aba957-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21eca854-5ee5-499d-9dd4-67d5a1aba957" (UID: "21eca854-5ee5-499d-9dd4-67d5a1aba957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:37:13.290424 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.290384 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21eca854-5ee5-499d-9dd4-67d5a1aba957-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:13.293267 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.293221 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21eca854-5ee5-499d-9dd4-67d5a1aba957-kube-api-access-4q8ms" (OuterVolumeSpecName: "kube-api-access-4q8ms") pod "21eca854-5ee5-499d-9dd4-67d5a1aba957" (UID: "21eca854-5ee5-499d-9dd4-67d5a1aba957"). InnerVolumeSpecName "kube-api-access-4q8ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:37:13.309108 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.309075 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr"] Sep 10 19:37:13.309365 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.309333 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" containerID="cri-o://70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589" gracePeriod=30 Sep 10 19:37:13.391869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:13.391828 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q8ms\" (UniqueName: \"kubernetes.io/projected/21eca854-5ee5-499d-9dd4-67d5a1aba957-kube-api-access-4q8ms\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:14.104004 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.103964 2584 generic.go:356] "Generic (PLEG): container finished" podID="21eca854-5ee5-499d-9dd4-67d5a1aba957" containerID="6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227" exitCode=0 Sep 10 19:37:14.104530 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.104015 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" Sep 10 19:37:14.104530 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.104051 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" event={"ID":"21eca854-5ee5-499d-9dd4-67d5a1aba957","Type":"ContainerDied","Data":"6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227"} Sep 10 19:37:14.104530 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.104122 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht" event={"ID":"21eca854-5ee5-499d-9dd4-67d5a1aba957","Type":"ContainerDied","Data":"39140e94bfed09d489d2f5789f8fabb14f3c0bc8161c7b357ad91ee26747d581"} Sep 10 19:37:14.104530 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.104159 2584 scope.go:117] "RemoveContainer" containerID="6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227" Sep 10 19:37:14.113142 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.113118 2584 scope.go:117] "RemoveContainer" containerID="6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227" Sep 10 19:37:14.113505 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:37:14.113485 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227\": container with ID starting with 6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227 not found: ID does not exist" containerID="6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227" Sep 10 19:37:14.113562 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.113517 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227"} err="failed to get container status \"6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227\": rpc error: code = NotFound desc = could not find container \"6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227\": container with ID starting with 6c685a240bdec2605c4acb0b0ee72573b0fa29467e3630f0859e4ff5f5786227 not found: ID does not exist" Sep 10 19:37:14.133512 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.133474 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht"] Sep 10 19:37:14.137223 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.137191 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cdcbd691-7f459f8d54-ntnht"] Sep 10 19:37:14.488211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.488136 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21eca854-5ee5-499d-9dd4-67d5a1aba957" path="/var/lib/kubelet/pods/21eca854-5ee5-499d-9dd4-67d5a1aba957/volumes" Sep 10 19:37:14.488522 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:14.488506 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6167b1d9-438e-4987-9650-d3690d97572a" path="/var/lib/kubelet/pods/6167b1d9-438e-4987-9650-d3690d97572a/volumes" Sep 10 19:37:17.454305 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:17.454233 2584 patch_prober.go:28] interesting pod/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 19:37:17.454777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:17.454320 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 19:37:18.567370 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.567340 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:37:18.642555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642453 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-credential-socket\") pod \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " Sep 10 19:37:18.642555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642531 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-podinfo\") pod \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " Sep 10 19:37:18.642555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642555 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istiod-ca-cert\") pod \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " Sep 10 19:37:18.642796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642605 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-token\") pod \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " Sep 10 19:37:18.642796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642681 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-certs\") pod \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " Sep 10 19:37:18.642796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642703 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-data\") pod \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " Sep 10 19:37:18.642796 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642731 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkhzz\" (UniqueName: \"kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-kube-api-access-xkhzz\") pod \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " Sep 10 19:37:18.643145 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.643121 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-socket\") pod \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " Sep 10 19:37:18.643303 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.643152 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-envoy\") pod \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\" (UID: \"7a276e15-eaf4-4d86-9600-9ee3c353f2ef\") " Sep 10 19:37:18.643303 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642756 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a276e15-eaf4-4d86-9600-9ee3c353f2ef" (UID: "7a276e15-eaf4-4d86-9600-9ee3c353f2ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:18.643303 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642975 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a276e15-eaf4-4d86-9600-9ee3c353f2ef" (UID: "7a276e15-eaf4-4d86-9600-9ee3c353f2ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:18.643303 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642993 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a276e15-eaf4-4d86-9600-9ee3c353f2ef" (UID: "7a276e15-eaf4-4d86-9600-9ee3c353f2ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:18.643303 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.642945 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a276e15-eaf4-4d86-9600-9ee3c353f2ef" (UID: "7a276e15-eaf4-4d86-9600-9ee3c353f2ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:37:18.643549 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.643366 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a276e15-eaf4-4d86-9600-9ee3c353f2ef" (UID: "7a276e15-eaf4-4d86-9600-9ee3c353f2ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:18.643607 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.643559 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:18.643607 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.643578 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-data\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:18.643607 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.643591 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-workload-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:18.643607 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.643605 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-credential-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:18.643807 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.643620 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istiod-ca-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:18.644863 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.644836 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a276e15-eaf4-4d86-9600-9ee3c353f2ef" (UID: "7a276e15-eaf4-4d86-9600-9ee3c353f2ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:37:18.644960 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.644860 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-kube-api-access-xkhzz" (OuterVolumeSpecName: "kube-api-access-xkhzz") pod "7a276e15-eaf4-4d86-9600-9ee3c353f2ef" (UID: "7a276e15-eaf4-4d86-9600-9ee3c353f2ef"). InnerVolumeSpecName "kube-api-access-xkhzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:37:18.644960 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.644877 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a276e15-eaf4-4d86-9600-9ee3c353f2ef" (UID: "7a276e15-eaf4-4d86-9600-9ee3c353f2ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:37:18.645108 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.645094 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a276e15-eaf4-4d86-9600-9ee3c353f2ef" (UID: "7a276e15-eaf4-4d86-9600-9ee3c353f2ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:18.744663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.744624 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-token\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:18.744663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.744656 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhzz\" (UniqueName: \"kubernetes.io/projected/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-kube-api-access-xkhzz\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:18.744663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.744666 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-envoy\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:18.744892 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:18.744688 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a276e15-eaf4-4d86-9600-9ee3c353f2ef-istio-podinfo\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:19.131083 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.131046 2584 generic.go:356] "Generic (PLEG): container finished" podID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerID="70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589" exitCode=0 Sep 10 19:37:19.131257 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.131093 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" Sep 10 19:37:19.131257 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.131137 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" event={"ID":"7a276e15-eaf4-4d86-9600-9ee3c353f2ef","Type":"ContainerDied","Data":"70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589"} Sep 10 19:37:19.131257 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.131189 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr" event={"ID":"7a276e15-eaf4-4d86-9600-9ee3c353f2ef","Type":"ContainerDied","Data":"a2df0915939fad0e3fb97602824a80e4eb5e4ce26023ba5aecfb75827b6b0da4"} Sep 10 19:37:19.131257 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.131214 2584 scope.go:117] "RemoveContainer" containerID="70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589" Sep 10 19:37:19.140567 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.140543 2584 scope.go:117] "RemoveContainer" containerID="0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c" Sep 10 19:37:19.148950 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.148924 2584 scope.go:117] "RemoveContainer" containerID="70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589" Sep 10 19:37:19.149308 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:37:19.149265 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589\": container with ID starting with 70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589 not found: ID does not exist" containerID="70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589" Sep 10 19:37:19.149379 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.149316 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589"} err="failed to get container status \"70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589\": rpc error: code = NotFound desc = could not find container \"70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589\": container with ID starting with 70d2e67fcfac5ab2784cb0237f7dbd74e07e42426003454c7402d3c118ce9589 not found: ID does not exist" Sep 10 19:37:19.149379 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.149337 2584 scope.go:117] "RemoveContainer" containerID="0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c" Sep 10 19:37:19.149609 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:37:19.149594 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c\": container with ID starting with 0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c not found: ID does not exist" containerID="0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c" Sep 10 19:37:19.149659 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.149613 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c"} err="failed to get container status \"0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c\": rpc error: code = NotFound desc = could not find container \"0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c\": container with ID starting with 0ee40e3efcf0288e4a49aa8dfcc60e1716b15c3239bf5d7155871354a7060c9c not found: ID does not exist" Sep 10 19:37:19.152916 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.152891 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr"] Sep 10 19:37:19.160252 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:19.160225 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-cdcbd691-openshift-default-7fc68666c6-v4ngr"] Sep 10 19:37:20.488322 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:20.488265 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" path="/var/lib/kubelet/pods/7a276e15-eaf4-4d86-9600-9ee3c353f2ef/volumes" Sep 10 19:37:34.261129 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261086 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf"] Sep 10 19:37:34.261528 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261501 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" Sep 10 19:37:34.261528 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261515 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" Sep 10 19:37:34.261528 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261529 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6167b1d9-438e-4987-9650-d3690d97572a" containerName="revoke-keys" Sep 10 19:37:34.261642 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261536 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6167b1d9-438e-4987-9650-d3690d97572a" containerName="revoke-keys" Sep 10 19:37:34.261642 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261549 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" Sep 10 19:37:34.261642 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261554 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" Sep 10 19:37:34.261642 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261576 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21eca854-5ee5-499d-9dd4-67d5a1aba957" containerName="payload-pre-processing" Sep 10 19:37:34.261642 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261581 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="21eca854-5ee5-499d-9dd4-67d5a1aba957" containerName="payload-pre-processing" Sep 10 19:37:34.261642 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261635 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="21eca854-5ee5-499d-9dd4-67d5a1aba957" containerName="payload-pre-processing" Sep 10 19:37:34.261818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261651 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" Sep 10 19:37:34.261818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261659 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6167b1d9-438e-4987-9650-d3690d97572a" containerName="revoke-keys" Sep 10 19:37:34.261818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.261665 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a276e15-eaf4-4d86-9600-9ee3c353f2ef" containerName="istio-proxy" Sep 10 19:37:34.269560 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.269529 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" Sep 10 19:37:34.272574 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.272539 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf"] Sep 10 19:37:34.273095 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.273064 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zztzd\"" Sep 10 19:37:34.273549 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.273473 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:37:34.394042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.393997 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56389931-e31a-48fb-addc-a03d7aaceb5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf\" (UID: \"56389931-e31a-48fb-addc-a03d7aaceb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" Sep 10 19:37:34.494626 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.494588 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56389931-e31a-48fb-addc-a03d7aaceb5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf\" (UID: \"56389931-e31a-48fb-addc-a03d7aaceb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" Sep 10 19:37:34.495175 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.495156 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56389931-e31a-48fb-addc-a03d7aaceb5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf\" (UID: \"56389931-e31a-48fb-addc-a03d7aaceb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" Sep 10 19:37:34.582347 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.582274 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" Sep 10 19:37:34.854409 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:34.854319 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf"] Sep 10 19:37:34.856773 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:37:34.856739 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56389931_e31a_48fb_addc_a03d7aaceb5a.slice/crio-b180138192434092891065944daa31f3151affb3cdc816fdf541b22a5b4de405 WatchSource:0}: Error finding container b180138192434092891065944daa31f3151affb3cdc816fdf541b22a5b4de405: Status 404 returned error can't find the container with id b180138192434092891065944daa31f3151affb3cdc816fdf541b22a5b4de405 Sep 10 19:37:35.217888 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:35.217849 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" event={"ID":"56389931-e31a-48fb-addc-a03d7aaceb5a","Type":"ContainerStarted","Data":"77538296e428538cebdb2a67e45d9da3cb95aa8a6da62fb66567284f2a5c8f36"} Sep 10 19:37:35.218077 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:35.217913 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" event={"ID":"56389931-e31a-48fb-addc-a03d7aaceb5a","Type":"ContainerStarted","Data":"b180138192434092891065944daa31f3151affb3cdc816fdf541b22a5b4de405"} Sep 10 19:37:35.235115 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:35.235050 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" podStartSLOduration=1.235032284 podStartE2EDuration="1.235032284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:37:35.232574755 +0000 UTC m=+1041.200387003" watchObservedRunningTime="2026-09-10 19:37:35.235032284 +0000 UTC m=+1041.202844530" Sep 10 19:37:36.224532 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:36.224493 2584 generic.go:356] "Generic (PLEG): container finished" podID="56389931-e31a-48fb-addc-a03d7aaceb5a" containerID="77538296e428538cebdb2a67e45d9da3cb95aa8a6da62fb66567284f2a5c8f36" exitCode=0 Sep 10 19:37:36.224965 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:36.224575 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" event={"ID":"56389931-e31a-48fb-addc-a03d7aaceb5a","Type":"ContainerDied","Data":"77538296e428538cebdb2a67e45d9da3cb95aa8a6da62fb66567284f2a5c8f36"} Sep 10 19:37:37.364314 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:37.364263 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" Sep 10 19:37:37.527181 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:37.527072 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56389931-e31a-48fb-addc-a03d7aaceb5a-maas-api-service-ca\") pod \"56389931-e31a-48fb-addc-a03d7aaceb5a\" (UID: \"56389931-e31a-48fb-addc-a03d7aaceb5a\") " Sep 10 19:37:37.527502 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:37.527476 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56389931-e31a-48fb-addc-a03d7aaceb5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56389931-e31a-48fb-addc-a03d7aaceb5a" (UID: "56389931-e31a-48fb-addc-a03d7aaceb5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:37:37.628782 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:37.628729 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56389931-e31a-48fb-addc-a03d7aaceb5a-maas-api-service-ca\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:37:38.235908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:38.235876 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" Sep 10 19:37:38.236128 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:38.235915 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf" event={"ID":"56389931-e31a-48fb-addc-a03d7aaceb5a","Type":"ContainerDied","Data":"b180138192434092891065944daa31f3151affb3cdc816fdf541b22a5b4de405"} Sep 10 19:37:38.236128 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:38.235966 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b180138192434092891065944daa31f3151affb3cdc816fdf541b22a5b4de405" Sep 10 19:37:44.930982 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:44.930945 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf"] Sep 10 19:37:44.933881 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:44.933851 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9c8a-fhqcf"] Sep 10 19:37:46.488613 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:37:46.488575 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56389931-e31a-48fb-addc-a03d7aaceb5a" path="/var/lib/kubelet/pods/56389931-e31a-48fb-addc-a03d7aaceb5a/volumes" Sep 10 19:38:39.897732 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:39.897696 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb"] Sep 10 19:38:39.898297 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:39.897913 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" podUID="f4fdbb8b-66f5-4d74-a704-6581f1771709" containerName="payload-processing" containerID="cri-o://04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba" gracePeriod=30 Sep 10 19:38:39.903292 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:39.903242 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt"] Sep 10 19:38:39.903735 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:39.903700 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" podUID="b1d8a473-18f0-4235-bcf3-570f227b5df7" containerName="payload-pre-processing" containerID="cri-o://11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8" gracePeriod=30 Sep 10 19:38:40.196709 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.196671 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:38:40.232656 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.232599 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:38:40.299669 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.299632 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fdbb8b-66f5-4d74-a704-6581f1771709-plugins-config-volume\") pod \"f4fdbb8b-66f5-4d74-a704-6581f1771709\" (UID: \"f4fdbb8b-66f5-4d74-a704-6581f1771709\") " Sep 10 19:38:40.299845 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.299706 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbxzq\" (UniqueName: \"kubernetes.io/projected/f4fdbb8b-66f5-4d74-a704-6581f1771709-kube-api-access-tbxzq\") pod \"f4fdbb8b-66f5-4d74-a704-6581f1771709\" (UID: \"f4fdbb8b-66f5-4d74-a704-6581f1771709\") " Sep 10 19:38:40.299845 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.299774 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wqrm\" (UniqueName: \"kubernetes.io/projected/b1d8a473-18f0-4235-bcf3-570f227b5df7-kube-api-access-6wqrm\") pod \"b1d8a473-18f0-4235-bcf3-570f227b5df7\" (UID: \"b1d8a473-18f0-4235-bcf3-570f227b5df7\") " Sep 10 19:38:40.299845 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.299812 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d8a473-18f0-4235-bcf3-570f227b5df7-plugins-config-volume\") pod \"b1d8a473-18f0-4235-bcf3-570f227b5df7\" (UID: \"b1d8a473-18f0-4235-bcf3-570f227b5df7\") " Sep 10 19:38:40.300228 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.300206 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d8a473-18f0-4235-bcf3-570f227b5df7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1d8a473-18f0-4235-bcf3-570f227b5df7" (UID: "b1d8a473-18f0-4235-bcf3-570f227b5df7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:38:40.300311 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.300207 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fdbb8b-66f5-4d74-a704-6581f1771709-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4fdbb8b-66f5-4d74-a704-6581f1771709" (UID: "f4fdbb8b-66f5-4d74-a704-6581f1771709"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:38:40.301940 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.301912 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fdbb8b-66f5-4d74-a704-6581f1771709-kube-api-access-tbxzq" (OuterVolumeSpecName: "kube-api-access-tbxzq") pod "f4fdbb8b-66f5-4d74-a704-6581f1771709" (UID: "f4fdbb8b-66f5-4d74-a704-6581f1771709"). InnerVolumeSpecName "kube-api-access-tbxzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:38:40.302046 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.301939 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d8a473-18f0-4235-bcf3-570f227b5df7-kube-api-access-6wqrm" (OuterVolumeSpecName: "kube-api-access-6wqrm") pod "b1d8a473-18f0-4235-bcf3-570f227b5df7" (UID: "b1d8a473-18f0-4235-bcf3-570f227b5df7"). InnerVolumeSpecName "kube-api-access-6wqrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:38:40.401112 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.401073 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wqrm\" (UniqueName: \"kubernetes.io/projected/b1d8a473-18f0-4235-bcf3-570f227b5df7-kube-api-access-6wqrm\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:40.401112 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.401104 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d8a473-18f0-4235-bcf3-570f227b5df7-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:40.401346 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.401114 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fdbb8b-66f5-4d74-a704-6581f1771709-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:40.401346 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.401139 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbxzq\" (UniqueName: \"kubernetes.io/projected/f4fdbb8b-66f5-4d74-a704-6581f1771709-kube-api-access-tbxzq\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:40.552606 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.552568 2584 generic.go:356] "Generic (PLEG): container finished" podID="f4fdbb8b-66f5-4d74-a704-6581f1771709" containerID="04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba" exitCode=0 Sep 10 19:38:40.552814 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.552629 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" Sep 10 19:38:40.552814 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.552648 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" event={"ID":"f4fdbb8b-66f5-4d74-a704-6581f1771709","Type":"ContainerDied","Data":"04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba"} Sep 10 19:38:40.552814 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.552712 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb" event={"ID":"f4fdbb8b-66f5-4d74-a704-6581f1771709","Type":"ContainerDied","Data":"7a7e49c21a8931b2c85dc4044ee3c19bfe34a1cb47028b5764936742b92f6bf0"} Sep 10 19:38:40.552814 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.552744 2584 scope.go:117] "RemoveContainer" containerID="04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba" Sep 10 19:38:40.554111 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.554087 2584 generic.go:356] "Generic (PLEG): container finished" podID="b1d8a473-18f0-4235-bcf3-570f227b5df7" containerID="11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8" exitCode=0 Sep 10 19:38:40.554217 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.554133 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" Sep 10 19:38:40.554217 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.554154 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" event={"ID":"b1d8a473-18f0-4235-bcf3-570f227b5df7","Type":"ContainerDied","Data":"11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8"} Sep 10 19:38:40.554217 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.554191 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt" event={"ID":"b1d8a473-18f0-4235-bcf3-570f227b5df7","Type":"ContainerDied","Data":"766d8a4aacde8ce61d6616787df2ff72e5d6ce147efeacd35f97f1181edb2175"} Sep 10 19:38:40.561745 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.561727 2584 scope.go:117] "RemoveContainer" containerID="04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba" Sep 10 19:38:40.562074 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:38:40.562052 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba\": container with ID starting with 04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba not found: ID does not exist" containerID="04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba" Sep 10 19:38:40.562182 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.562084 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba"} err="failed to get container status \"04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba\": rpc error: code = NotFound desc = could not find container \"04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba\": container with ID starting with 04fc25288403f0c6e527f2b86ee9e43d18c28ee3836a2c714e857674abf43bba not found: ID does not exist" Sep 10 19:38:40.562182 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.562101 2584 scope.go:117] "RemoveContainer" containerID="11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8" Sep 10 19:38:40.569669 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.569640 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt"] Sep 10 19:38:40.570651 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.570631 2584 scope.go:117] "RemoveContainer" containerID="11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8" Sep 10 19:38:40.570977 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:38:40.570956 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8\": container with ID starting with 11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8 not found: ID does not exist" containerID="11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8" Sep 10 19:38:40.571044 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.570995 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8"} err="failed to get container status \"11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8\": rpc error: code = NotFound desc = could not find container \"11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8\": container with ID starting with 11fb4b9f74bbaaca1c77a00f8d645c2ca2d7b2b6176d7385c1a52d660e0604d8 not found: ID does not exist" Sep 10 19:38:40.571563 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.571541 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1590-7f69d66cbb-khtpt"] Sep 10 19:38:40.581702 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.581670 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb"] Sep 10 19:38:40.585886 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:40.585854 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1590-68b74b4467-9h8gb"] Sep 10 19:38:42.488439 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:42.488404 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1d8a473-18f0-4235-bcf3-570f227b5df7" path="/var/lib/kubelet/pods/b1d8a473-18f0-4235-bcf3-570f227b5df7/volumes" Sep 10 19:38:42.488836 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:42.488789 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4fdbb8b-66f5-4d74-a704-6581f1771709" path="/var/lib/kubelet/pods/f4fdbb8b-66f5-4d74-a704-6581f1771709/volumes" Sep 10 19:38:46.963072 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:46.963034 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk"] Sep 10 19:38:46.963688 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:46.963326 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" containerID="cri-o://396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955" gracePeriod=30 Sep 10 19:38:52.218806 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.218774 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:38:52.314362 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314322 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-socket\") pod \"e6440fe6-251e-4ea5-94be-4f25951985ab\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " Sep 10 19:38:52.314552 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314378 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-data\") pod \"e6440fe6-251e-4ea5-94be-4f25951985ab\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " Sep 10 19:38:52.314552 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314403 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddj8t\" (UniqueName: \"kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-kube-api-access-ddj8t\") pod \"e6440fe6-251e-4ea5-94be-4f25951985ab\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " Sep 10 19:38:52.314552 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314453 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-token\") pod \"e6440fe6-251e-4ea5-94be-4f25951985ab\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " Sep 10 19:38:52.314552 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314478 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-certs\") pod \"e6440fe6-251e-4ea5-94be-4f25951985ab\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " Sep 10 19:38:52.314777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314570 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6440fe6-251e-4ea5-94be-4f25951985ab-istiod-ca-cert\") pod \"e6440fe6-251e-4ea5-94be-4f25951985ab\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " Sep 10 19:38:52.314777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314595 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-credential-socket\") pod \"e6440fe6-251e-4ea5-94be-4f25951985ab\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " Sep 10 19:38:52.314777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314661 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-envoy\") pod \"e6440fe6-251e-4ea5-94be-4f25951985ab\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " Sep 10 19:38:52.314777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314663 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "e6440fe6-251e-4ea5-94be-4f25951985ab" (UID: "e6440fe6-251e-4ea5-94be-4f25951985ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:38:52.314777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314658 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e6440fe6-251e-4ea5-94be-4f25951985ab" (UID: "e6440fe6-251e-4ea5-94be-4f25951985ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:38:52.314777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314713 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-podinfo\") pod \"e6440fe6-251e-4ea5-94be-4f25951985ab\" (UID: \"e6440fe6-251e-4ea5-94be-4f25951985ab\") " Sep 10 19:38:52.315078 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314796 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e6440fe6-251e-4ea5-94be-4f25951985ab" (UID: "e6440fe6-251e-4ea5-94be-4f25951985ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:38:52.315078 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314891 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e6440fe6-251e-4ea5-94be-4f25951985ab" (UID: "e6440fe6-251e-4ea5-94be-4f25951985ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:38:52.315078 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.314976 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6440fe6-251e-4ea5-94be-4f25951985ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e6440fe6-251e-4ea5-94be-4f25951985ab" (UID: "e6440fe6-251e-4ea5-94be-4f25951985ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:38:52.315078 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.315036 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6440fe6-251e-4ea5-94be-4f25951985ab-istiod-ca-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.315078 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.315055 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-credential-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.315078 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.315070 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.315426 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.315088 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-data\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.315426 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.315106 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-workload-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.316924 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.316897 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "e6440fe6-251e-4ea5-94be-4f25951985ab" (UID: "e6440fe6-251e-4ea5-94be-4f25951985ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:38:52.317044 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.316898 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e6440fe6-251e-4ea5-94be-4f25951985ab" (UID: "e6440fe6-251e-4ea5-94be-4f25951985ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:38:52.317092 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.317056 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-kube-api-access-ddj8t" (OuterVolumeSpecName: "kube-api-access-ddj8t") pod "e6440fe6-251e-4ea5-94be-4f25951985ab" (UID: "e6440fe6-251e-4ea5-94be-4f25951985ab"). InnerVolumeSpecName "kube-api-access-ddj8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:38:52.317206 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.317188 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e6440fe6-251e-4ea5-94be-4f25951985ab" (UID: "e6440fe6-251e-4ea5-94be-4f25951985ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:38:52.416362 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.416326 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-envoy\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.416362 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.416360 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-podinfo\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.416587 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.416373 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddj8t\" (UniqueName: \"kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-kube-api-access-ddj8t\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.416587 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.416386 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6440fe6-251e-4ea5-94be-4f25951985ab-istio-token\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.618525 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.618391 2584 generic.go:356] "Generic (PLEG): container finished" podID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerID="396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955" exitCode=0 Sep 10 19:38:52.618677 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.618536 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" Sep 10 19:38:52.618677 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.618546 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" event={"ID":"e6440fe6-251e-4ea5-94be-4f25951985ab","Type":"ContainerDied","Data":"396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955"} Sep 10 19:38:52.618677 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.618617 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk" event={"ID":"e6440fe6-251e-4ea5-94be-4f25951985ab","Type":"ContainerDied","Data":"dec096185736d2896a7084ce0492111c6892aa13e48408256979af445059705b"} Sep 10 19:38:52.618677 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.618642 2584 scope.go:117] "RemoveContainer" containerID="396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955" Sep 10 19:38:52.627571 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.627551 2584 scope.go:117] "RemoveContainer" containerID="a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd" Sep 10 19:38:52.637164 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.637092 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk"] Sep 10 19:38:52.638355 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.638320 2584 scope.go:117] "RemoveContainer" containerID="396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955" Sep 10 19:38:52.639808 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:38:52.639489 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955\": container with ID starting with 396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955 not found: ID does not exist" containerID="396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955" Sep 10 19:38:52.639808 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.639530 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955"} err="failed to get container status \"396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955\": rpc error: code = NotFound desc = could not find container \"396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955\": container with ID starting with 396601cf2e3d86c4c1aee7c6a041400c8cbfaf4e7af0d7b89d93deadb2c09955 not found: ID does not exist" Sep 10 19:38:52.639808 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.639556 2584 scope.go:117] "RemoveContainer" containerID="a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd" Sep 10 19:38:52.640043 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:38:52.639891 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd\": container with ID starting with a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd not found: ID does not exist" containerID="a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd" Sep 10 19:38:52.640043 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.639921 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd"} err="failed to get container status \"a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd\": rpc error: code = NotFound desc = could not find container \"a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd\": container with ID starting with a60bb9bf4d506c419a94154d85b15de77109c23898e7d8379dcc905dd7fb31dd not found: ID does not exist" Sep 10 19:38:52.640818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:52.640796 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3a1590-openshift-default-5764cb7c84-689xk"] Sep 10 19:38:54.488931 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:54.488898 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" path="/var/lib/kubelet/pods/e6440fe6-251e-4ea5-94be-4f25951985ab/volumes" Sep 10 19:38:55.380683 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.380641 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p"] Sep 10 19:38:55.381134 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381114 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4fdbb8b-66f5-4d74-a704-6581f1771709" containerName="payload-processing" Sep 10 19:38:55.381229 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381136 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4fdbb8b-66f5-4d74-a704-6581f1771709" containerName="payload-processing" Sep 10 19:38:55.381229 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381146 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" Sep 10 19:38:55.381229 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381155 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" Sep 10 19:38:55.381229 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381165 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" Sep 10 19:38:55.381229 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381174 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" Sep 10 19:38:55.381229 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381192 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d8a473-18f0-4235-bcf3-570f227b5df7" containerName="payload-pre-processing" Sep 10 19:38:55.381229 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381201 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d8a473-18f0-4235-bcf3-570f227b5df7" containerName="payload-pre-processing" Sep 10 19:38:55.381229 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381226 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56389931-e31a-48fb-addc-a03d7aaceb5a" containerName="revoke-keys" Sep 10 19:38:55.381707 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381235 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="56389931-e31a-48fb-addc-a03d7aaceb5a" containerName="revoke-keys" Sep 10 19:38:55.381707 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381356 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d8a473-18f0-4235-bcf3-570f227b5df7" containerName="payload-pre-processing" Sep 10 19:38:55.381707 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381373 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4fdbb8b-66f5-4d74-a704-6581f1771709" containerName="payload-processing" Sep 10 19:38:55.381707 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381384 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" Sep 10 19:38:55.381707 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381393 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="56389931-e31a-48fb-addc-a03d7aaceb5a" containerName="revoke-keys" Sep 10 19:38:55.381707 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.381402 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6440fe6-251e-4ea5-94be-4f25951985ab" containerName="istio-proxy" Sep 10 19:38:55.386324 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.386298 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:38:55.388956 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.388928 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f5248185-dockercfg-d5z8v\"" Sep 10 19:38:55.389315 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.389293 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f5248185\"" Sep 10 19:38:55.402544 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.402513 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p"] Sep 10 19:38:55.445245 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.445199 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5fj\" (UniqueName: \"kubernetes.io/projected/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-kube-api-access-qj5fj\") pod \"payload-processing-e2e-mt-f5248185-76877949b9-rx72p\" (UID: \"bf0b8bb7-b61b-4952-aea4-90d62b24aa04\") " pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:38:55.445452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.445339 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-plugins-config-volume\") pod \"payload-processing-e2e-mt-f5248185-76877949b9-rx72p\" (UID: \"bf0b8bb7-b61b-4952-aea4-90d62b24aa04\") " pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:38:55.546873 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.546830 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5fj\" (UniqueName: \"kubernetes.io/projected/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-kube-api-access-qj5fj\") pod \"payload-processing-e2e-mt-f5248185-76877949b9-rx72p\" (UID: \"bf0b8bb7-b61b-4952-aea4-90d62b24aa04\") " pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:38:55.547425 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.546906 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-plugins-config-volume\") pod \"payload-processing-e2e-mt-f5248185-76877949b9-rx72p\" (UID: \"bf0b8bb7-b61b-4952-aea4-90d62b24aa04\") " pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:38:55.547772 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.547746 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-plugins-config-volume\") pod \"payload-processing-e2e-mt-f5248185-76877949b9-rx72p\" (UID: \"bf0b8bb7-b61b-4952-aea4-90d62b24aa04\") " pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:38:55.554790 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.554762 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5fj\" (UniqueName: \"kubernetes.io/projected/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-kube-api-access-qj5fj\") pod \"payload-processing-e2e-mt-f5248185-76877949b9-rx72p\" (UID: \"bf0b8bb7-b61b-4952-aea4-90d62b24aa04\") " pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:38:55.697131 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.697035 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:38:55.974901 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:55.974816 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p"] Sep 10 19:38:55.978135 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:38:55.978106 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0b8bb7_b61b_4952_aea4_90d62b24aa04.slice/crio-d1090c1d1733a0ca43a845a3ad2c0bfe699751a30035f7a4b8b509e0239300b8 WatchSource:0}: Error finding container d1090c1d1733a0ca43a845a3ad2c0bfe699751a30035f7a4b8b509e0239300b8: Status 404 returned error can't find the container with id d1090c1d1733a0ca43a845a3ad2c0bfe699751a30035f7a4b8b509e0239300b8 Sep 10 19:38:56.639826 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:56.639789 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" event={"ID":"bf0b8bb7-b61b-4952-aea4-90d62b24aa04","Type":"ContainerStarted","Data":"f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7"} Sep 10 19:38:56.640216 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:56.639847 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" event={"ID":"bf0b8bb7-b61b-4952-aea4-90d62b24aa04","Type":"ContainerStarted","Data":"d1090c1d1733a0ca43a845a3ad2c0bfe699751a30035f7a4b8b509e0239300b8"} Sep 10 19:38:56.640216 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:56.639889 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:38:56.658441 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:56.658380 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" podStartSLOduration=1.658359881 podStartE2EDuration="1.658359881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:38:56.658178044 +0000 UTC m=+1122.625990290" watchObservedRunningTime="2026-09-10 19:38:56.658359881 +0000 UTC m=+1122.626172128" Sep 10 19:38:58.128251 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.128210 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj"] Sep 10 19:38:58.128743 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.128537 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" podUID="e322e040-cc76-4925-8171-ba15d9567bc5" containerName="payload-processing" containerID="cri-o://9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1" gracePeriod=30 Sep 10 19:38:58.387800 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.387726 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:38:58.475645 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.475609 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6whhh\" (UniqueName: \"kubernetes.io/projected/e322e040-cc76-4925-8171-ba15d9567bc5-kube-api-access-6whhh\") pod \"e322e040-cc76-4925-8171-ba15d9567bc5\" (UID: \"e322e040-cc76-4925-8171-ba15d9567bc5\") " Sep 10 19:38:58.475908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.475666 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e322e040-cc76-4925-8171-ba15d9567bc5-plugins-config-volume\") pod \"e322e040-cc76-4925-8171-ba15d9567bc5\" (UID: \"e322e040-cc76-4925-8171-ba15d9567bc5\") " Sep 10 19:38:58.476251 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.476217 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e322e040-cc76-4925-8171-ba15d9567bc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e322e040-cc76-4925-8171-ba15d9567bc5" (UID: "e322e040-cc76-4925-8171-ba15d9567bc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:38:58.478450 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.478425 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e322e040-cc76-4925-8171-ba15d9567bc5-kube-api-access-6whhh" (OuterVolumeSpecName: "kube-api-access-6whhh") pod "e322e040-cc76-4925-8171-ba15d9567bc5" (UID: "e322e040-cc76-4925-8171-ba15d9567bc5"). InnerVolumeSpecName "kube-api-access-6whhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:38:58.576817 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.576777 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6whhh\" (UniqueName: \"kubernetes.io/projected/e322e040-cc76-4925-8171-ba15d9567bc5-kube-api-access-6whhh\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:58.576817 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.576810 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e322e040-cc76-4925-8171-ba15d9567bc5-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:38:58.651329 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.651208 2584 generic.go:356] "Generic (PLEG): container finished" podID="e322e040-cc76-4925-8171-ba15d9567bc5" containerID="9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1" exitCode=0 Sep 10 19:38:58.651329 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.651257 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" Sep 10 19:38:58.651329 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.651258 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" event={"ID":"e322e040-cc76-4925-8171-ba15d9567bc5","Type":"ContainerDied","Data":"9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1"} Sep 10 19:38:58.651573 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.651342 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj" event={"ID":"e322e040-cc76-4925-8171-ba15d9567bc5","Type":"ContainerDied","Data":"9a7c17f8d8463f11f881b5fd689984fa29b2382f09b9884cad2bd9a6d46e7e35"} Sep 10 19:38:58.651573 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.651374 2584 scope.go:117] "RemoveContainer" containerID="9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1" Sep 10 19:38:58.660083 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.660062 2584 scope.go:117] "RemoveContainer" containerID="9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1" Sep 10 19:38:58.660447 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:38:58.660427 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1\": container with ID starting with 9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1 not found: ID does not exist" containerID="9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1" Sep 10 19:38:58.660516 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.660456 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1"} err="failed to get container status \"9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1\": rpc error: code = NotFound desc = could not find container \"9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1\": container with ID starting with 9753ffacae95866fbab52bbd32532b6629e68bc78e26d9b4fcda0df4a4d832c1 not found: ID does not exist" Sep 10 19:38:58.669491 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.669459 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj"] Sep 10 19:38:58.673382 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:38:58.673351 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a037c8-5455df7779-hhrrj"] Sep 10 19:39:00.487999 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:00.487951 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e322e040-cc76-4925-8171-ba15d9567bc5" path="/var/lib/kubelet/pods/e322e040-cc76-4925-8171-ba15d9567bc5/volumes" Sep 10 19:39:05.746673 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.746637 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp"] Sep 10 19:39:05.747126 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.747107 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e322e040-cc76-4925-8171-ba15d9567bc5" containerName="payload-processing" Sep 10 19:39:05.747126 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.747128 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e322e040-cc76-4925-8171-ba15d9567bc5" containerName="payload-processing" Sep 10 19:39:05.747219 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.747210 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e322e040-cc76-4925-8171-ba15d9567bc5" containerName="payload-processing" Sep 10 19:39:05.752063 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.752026 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.754509 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.754484 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-07c5fe-openshift-default-dockercfg-lnhtf\"" Sep 10 19:39:05.764920 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.764887 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp"] Sep 10 19:39:05.840679 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.840641 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-data\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.840679 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.840682 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361212c6-d3d6-454f-ae24-f2882b1e3469-istiod-ca-cert\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.840985 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.840745 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-podinfo\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.840985 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.840792 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-token\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.840985 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.840833 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-envoy\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.840985 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.840886 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-socket\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.840985 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.840906 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm7v7\" (UniqueName: \"kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-kube-api-access-bm7v7\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.840985 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.840977 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-credential-socket\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.841317 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.841001 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-certs\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942432 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942390 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-credential-socket\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942432 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942429 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-certs\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942696 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942459 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-data\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942696 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942575 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361212c6-d3d6-454f-ae24-f2882b1e3469-istiod-ca-cert\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942696 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942638 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-podinfo\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942696 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942671 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-token\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942721 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-envoy\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942797 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-socket\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942820 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7v7\" (UniqueName: \"kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-kube-api-access-bm7v7\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942864 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-credential-socket\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.942905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942896 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-data\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.943182 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.942974 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-certs\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.943182 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.943140 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-socket\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.943314 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.943299 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361212c6-d3d6-454f-ae24-f2882b1e3469-istiod-ca-cert\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.945168 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.945141 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-envoy\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.945403 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.945382 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-podinfo\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.950981 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.950952 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-token\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:05.951098 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:05.951081 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm7v7\" (UniqueName: \"kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-kube-api-access-bm7v7\") pod \"e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:06.062374 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:06.062329 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:06.339567 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:06.339538 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp"] Sep 10 19:39:06.341464 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:39:06.341436 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361212c6_d3d6_454f_ae24_f2882b1e3469.slice/crio-f6e95a7ad27de3ca6636bb1684c0d08bd84a32df122936a0d53ce8b1fbab4bbf WatchSource:0}: Error finding container f6e95a7ad27de3ca6636bb1684c0d08bd84a32df122936a0d53ce8b1fbab4bbf: Status 404 returned error can't find the container with id f6e95a7ad27de3ca6636bb1684c0d08bd84a32df122936a0d53ce8b1fbab4bbf Sep 10 19:39:06.343895 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:06.343863 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:06.343965 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:06.343933 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:06.344003 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:06.343965 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:06.695831 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:06.695739 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" event={"ID":"361212c6-d3d6-454f-ae24-f2882b1e3469","Type":"ContainerStarted","Data":"f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34"} Sep 10 19:39:06.695831 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:06.695798 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" event={"ID":"361212c6-d3d6-454f-ae24-f2882b1e3469","Type":"ContainerStarted","Data":"f6e95a7ad27de3ca6636bb1684c0d08bd84a32df122936a0d53ce8b1fbab4bbf"} Sep 10 19:39:06.717947 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:06.717878 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" podStartSLOduration=1.717853603 podStartE2EDuration="1.717853603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:39:06.715650216 +0000 UTC m=+1132.683462475" watchObservedRunningTime="2026-09-10 19:39:06.717853603 +0000 UTC m=+1132.685665853" Sep 10 19:39:07.062723 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:07.062683 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:07.068012 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:07.067980 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:07.647765 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:07.647725 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:39:07.707590 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:07.707552 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:07.709721 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:07.709694 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:23.749484 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:23.749436 2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 19:39:23.749991 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:23.749524 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 19:39:25.597081 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.597040 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr"] Sep 10 19:39:25.599431 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.599413 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" Sep 10 19:39:25.601876 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.601852 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:39:25.602027 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.602009 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zztzd\"" Sep 10 19:39:25.608424 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.608393 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr"] Sep 10 19:39:25.634333 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.634294 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f5248185-pdllr\" (UID: \"7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" Sep 10 19:39:25.735029 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.734936 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f5248185-pdllr\" (UID: \"7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" Sep 10 19:39:25.735570 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.735550 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f5248185-pdllr\" (UID: \"7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" Sep 10 19:39:25.806730 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.806690 2584 generic.go:356] "Generic (PLEG): container finished" podID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerID="f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34" exitCode=0 Sep 10 19:39:25.806923 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.806796 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" event={"ID":"361212c6-d3d6-454f-ae24-f2882b1e3469","Type":"ContainerDied","Data":"f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34"} Sep 10 19:39:25.807385 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.807364 2584 scope.go:117] "RemoveContainer" containerID="f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34" Sep 10 19:39:25.808571 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.808536 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:25.808667 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.808633 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:25.808725 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.808679 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:25.911558 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:25.911517 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" Sep 10 19:39:26.196318 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:26.196254 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr"] Sep 10 19:39:26.197824 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:39:26.197790 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdfb97d_1b52_4a0d_b883_1d08a5f8cdaa.slice/crio-1cf4295821cd775230caa705c1f738ef52d91fa69377730516cb59465a23ee6c WatchSource:0}: Error finding container 1cf4295821cd775230caa705c1f738ef52d91fa69377730516cb59465a23ee6c: Status 404 returned error can't find the container with id 1cf4295821cd775230caa705c1f738ef52d91fa69377730516cb59465a23ee6c Sep 10 19:39:26.813599 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:26.813559 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" event={"ID":"7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa","Type":"ContainerStarted","Data":"718042b6b2c05ff4d0d2fdc83fcbeac376b5c9530c4954f8683d55155bc4a8a2"} Sep 10 19:39:26.814059 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:26.813625 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" event={"ID":"7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa","Type":"ContainerStarted","Data":"1cf4295821cd775230caa705c1f738ef52d91fa69377730516cb59465a23ee6c"} Sep 10 19:39:26.816023 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:26.815986 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" event={"ID":"361212c6-d3d6-454f-ae24-f2882b1e3469","Type":"ContainerStarted","Data":"f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f"} Sep 10 19:39:26.851098 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:26.851036 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" podStartSLOduration=1.851018912 podStartE2EDuration="1.851018912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:39:26.83120974 +0000 UTC m=+1152.799021988" watchObservedRunningTime="2026-09-10 19:39:26.851018912 +0000 UTC m=+1152.818831155" Sep 10 19:39:27.063163 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:27.063120 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:27.063473 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:27.063445 2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 10 19:39:27.063595 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:27.063508 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 10 19:39:27.833380 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:27.833328 2584 generic.go:356] "Generic (PLEG): container finished" podID="7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa" containerID="718042b6b2c05ff4d0d2fdc83fcbeac376b5c9530c4954f8683d55155bc4a8a2" exitCode=0 Sep 10 19:39:27.833876 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:27.833743 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" event={"ID":"7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa","Type":"ContainerDied","Data":"718042b6b2c05ff4d0d2fdc83fcbeac376b5c9530c4954f8683d55155bc4a8a2"} Sep 10 19:39:28.063812 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:28.063770 2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 10 19:39:28.063997 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:28.063834 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 10 19:39:28.991555 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:28.991525 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" Sep 10 19:39:29.063204 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:29.063163 2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 10 19:39:29.063412 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:29.063229 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 10 19:39:29.066529 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:29.066499 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa-maas-api-service-ca\") pod \"7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa\" (UID: \"7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa\") " Sep 10 19:39:29.066835 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:29.066812 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa" (UID: "7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:29.167445 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:29.167349 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa-maas-api-service-ca\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:39:29.846658 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:29.846624 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" Sep 10 19:39:29.846841 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:29.846664 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr" event={"ID":"7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa","Type":"ContainerDied","Data":"1cf4295821cd775230caa705c1f738ef52d91fa69377730516cb59465a23ee6c"} Sep 10 19:39:29.846841 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:29.846715 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cf4295821cd775230caa705c1f738ef52d91fa69377730516cb59465a23ee6c" Sep 10 19:39:30.063048 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:30.063005 2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 10 19:39:30.063472 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:30.063064 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 10 19:39:31.062751 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:31.062705 2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 10 19:39:31.062946 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:31.062768 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 10 19:39:32.067079 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:32.067043 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:32.067596 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:32.067327 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:32.068054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:32.068036 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:39:34.181873 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.181838 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh"] Sep 10 19:39:34.182398 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.182358 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa" containerName="revoke-keys" Sep 10 19:39:34.182398 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.182375 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa" containerName="revoke-keys" Sep 10 19:39:34.182540 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.182493 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa" containerName="revoke-keys" Sep 10 19:39:34.184656 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.184635 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.186865 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.186844 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-2a1fa9-openshift-default-dockercfg-9tcxs\"" Sep 10 19:39:34.198249 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.198219 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh"] Sep 10 19:39:34.318908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.318861 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-credential-socket\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.319096 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.318917 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-envoy\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.319096 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.319015 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmjvd\" (UniqueName: \"kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-kube-api-access-qmjvd\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.319096 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.319059 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-istio-token\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.319205 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.319155 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-certs\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.319205 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.319197 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-socket\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.319344 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.319257 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfac205e-0cb1-426c-8411-1dad1d224947-istio-podinfo\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.319344 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.319313 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-data\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.319344 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.319335 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfac205e-0cb1-426c-8411-1dad1d224947-istiod-ca-cert\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.420659 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.420617 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-istio-token\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.420860 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.420782 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-certs\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.420860 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.420832 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-socket\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.420988 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.420880 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfac205e-0cb1-426c-8411-1dad1d224947-istio-podinfo\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.420988 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.420910 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-data\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.420988 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.420936 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfac205e-0cb1-426c-8411-1dad1d224947-istiod-ca-cert\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.420988 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.420966 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-credential-socket\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.421189 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.421013 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-envoy\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.421189 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.421063 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmjvd\" (UniqueName: \"kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-kube-api-access-qmjvd\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.421311 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.421253 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-certs\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.421376 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.421344 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-data\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.421501 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.421476 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-socket\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.421612 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.421589 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-credential-socket\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.421735 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.421713 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfac205e-0cb1-426c-8411-1dad1d224947-istiod-ca-cert\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.423318 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.423269 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-envoy\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.423446 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.423427 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfac205e-0cb1-426c-8411-1dad1d224947-istio-podinfo\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.429452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.429420 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-istio-token\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.429714 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.429685 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmjvd\" (UniqueName: \"kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-kube-api-access-qmjvd\") pod \"e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.498273 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.498177 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:34.777689 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.777604 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh"] Sep 10 19:39:34.781055 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:39:34.781023 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfac205e_0cb1_426c_8411_1dad1d224947.slice/crio-357981a52602d358b568946f6c8077fc00ce85ac7c44632ac3afd41698127d20 WatchSource:0}: Error finding container 357981a52602d358b568946f6c8077fc00ce85ac7c44632ac3afd41698127d20: Status 404 returned error can't find the container with id 357981a52602d358b568946f6c8077fc00ce85ac7c44632ac3afd41698127d20 Sep 10 19:39:34.783326 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.783263 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:34.783398 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.783367 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:34.783448 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.783398 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:34.874535 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.874486 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" event={"ID":"cfac205e-0cb1-426c-8411-1dad1d224947","Type":"ContainerStarted","Data":"f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7"} Sep 10 19:39:34.874718 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.874574 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" event={"ID":"cfac205e-0cb1-426c-8411-1dad1d224947","Type":"ContainerStarted","Data":"357981a52602d358b568946f6c8077fc00ce85ac7c44632ac3afd41698127d20"} Sep 10 19:39:34.898753 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:34.898678 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" podStartSLOduration=0.898653101 podStartE2EDuration="898.653101ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:39:34.895392945 +0000 UTC m=+1160.863205193" watchObservedRunningTime="2026-09-10 19:39:34.898653101 +0000 UTC m=+1160.866465348" Sep 10 19:39:35.499078 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:35.499032 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:35.504220 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:35.504188 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:35.879320 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:35.879263 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:35.880519 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:35.880498 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:36.537713 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.537671 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p"] Sep 10 19:39:36.538087 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.537952 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" podUID="bf0b8bb7-b61b-4952-aea4-90d62b24aa04" containerName="payload-processing" containerID="cri-o://f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7" gracePeriod=30 Sep 10 19:39:36.727805 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.727757 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr"] Sep 10 19:39:36.731936 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.731899 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5248185-pdllr"] Sep 10 19:39:36.804460 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.804429 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:39:36.885708 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.885622 2584 generic.go:356] "Generic (PLEG): container finished" podID="bf0b8bb7-b61b-4952-aea4-90d62b24aa04" containerID="f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7" exitCode=0 Sep 10 19:39:36.885708 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.885665 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" Sep 10 19:39:36.885955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.885711 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" event={"ID":"bf0b8bb7-b61b-4952-aea4-90d62b24aa04","Type":"ContainerDied","Data":"f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7"} Sep 10 19:39:36.885955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.885766 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p" event={"ID":"bf0b8bb7-b61b-4952-aea4-90d62b24aa04","Type":"ContainerDied","Data":"d1090c1d1733a0ca43a845a3ad2c0bfe699751a30035f7a4b8b509e0239300b8"} Sep 10 19:39:36.885955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.885796 2584 scope.go:117] "RemoveContainer" containerID="f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7" Sep 10 19:39:36.894411 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.894386 2584 scope.go:117] "RemoveContainer" containerID="f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7" Sep 10 19:39:36.894756 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:39:36.894730 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7\": container with ID starting with f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7 not found: ID does not exist" containerID="f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7" Sep 10 19:39:36.894865 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.894764 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7"} err="failed to get container status \"f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7\": rpc error: code = NotFound desc = could not find container \"f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7\": container with ID starting with f9a9acd57eca3ccd38643c16849fa8ee548838cbd6817ce4a0af4db43ae791d7 not found: ID does not exist" Sep 10 19:39:36.946050 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.946012 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-plugins-config-volume\") pod \"bf0b8bb7-b61b-4952-aea4-90d62b24aa04\" (UID: \"bf0b8bb7-b61b-4952-aea4-90d62b24aa04\") " Sep 10 19:39:36.946204 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.946172 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj5fj\" (UniqueName: \"kubernetes.io/projected/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-kube-api-access-qj5fj\") pod \"bf0b8bb7-b61b-4952-aea4-90d62b24aa04\" (UID: \"bf0b8bb7-b61b-4952-aea4-90d62b24aa04\") " Sep 10 19:39:36.946527 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.946496 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf0b8bb7-b61b-4952-aea4-90d62b24aa04" (UID: "bf0b8bb7-b61b-4952-aea4-90d62b24aa04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:36.948395 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:36.948370 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-kube-api-access-qj5fj" (OuterVolumeSpecName: "kube-api-access-qj5fj") pod "bf0b8bb7-b61b-4952-aea4-90d62b24aa04" (UID: "bf0b8bb7-b61b-4952-aea4-90d62b24aa04"). InnerVolumeSpecName "kube-api-access-qj5fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:39:37.047955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:37.047912 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj5fj\" (UniqueName: \"kubernetes.io/projected/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-kube-api-access-qj5fj\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:39:37.047955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:37.047946 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0b8bb7-b61b-4952-aea4-90d62b24aa04-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:39:37.206907 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:37.206865 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p"] Sep 10 19:39:37.210736 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:37.210705 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f5248185-76877949b9-rx72p"] Sep 10 19:39:38.488436 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.488402 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa" path="/var/lib/kubelet/pods/7fdfb97d-1b52-4a0d-b883-1d08a5f8cdaa/volumes" Sep 10 19:39:38.488847 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.488833 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0b8bb7-b61b-4952-aea4-90d62b24aa04" path="/var/lib/kubelet/pods/bf0b8bb7-b61b-4952-aea4-90d62b24aa04/volumes" Sep 10 19:39:38.737470 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.737432 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4"] Sep 10 19:39:38.737793 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.737781 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0b8bb7-b61b-4952-aea4-90d62b24aa04" containerName="payload-processing" Sep 10 19:39:38.737838 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.737795 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0b8bb7-b61b-4952-aea4-90d62b24aa04" containerName="payload-processing" Sep 10 19:39:38.737882 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.737873 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0b8bb7-b61b-4952-aea4-90d62b24aa04" containerName="payload-processing" Sep 10 19:39:38.740898 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.740845 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:39:38.743135 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.743111 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-2a1fa9-dockercfg-lbv8t\"" Sep 10 19:39:38.743694 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.743674 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-2a1fa9\"" Sep 10 19:39:38.752513 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.752482 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4"] Sep 10 19:39:38.866480 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.866438 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87f0afe-5140-4636-a690-bbc13cf75d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4\" (UID: \"f87f0afe-5140-4636-a690-bbc13cf75d78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:39:38.866647 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.866490 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65rxg\" (UniqueName: \"kubernetes.io/projected/f87f0afe-5140-4636-a690-bbc13cf75d78-kube-api-access-65rxg\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4\" (UID: \"f87f0afe-5140-4636-a690-bbc13cf75d78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:39:38.968009 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.967969 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87f0afe-5140-4636-a690-bbc13cf75d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4\" (UID: \"f87f0afe-5140-4636-a690-bbc13cf75d78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:39:38.968009 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.968025 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65rxg\" (UniqueName: \"kubernetes.io/projected/f87f0afe-5140-4636-a690-bbc13cf75d78-kube-api-access-65rxg\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4\" (UID: \"f87f0afe-5140-4636-a690-bbc13cf75d78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:39:38.968831 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.968807 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87f0afe-5140-4636-a690-bbc13cf75d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4\" (UID: \"f87f0afe-5140-4636-a690-bbc13cf75d78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:39:38.976666 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:38.976630 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rxg\" (UniqueName: \"kubernetes.io/projected/f87f0afe-5140-4636-a690-bbc13cf75d78-kube-api-access-65rxg\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4\" (UID: \"f87f0afe-5140-4636-a690-bbc13cf75d78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:39:39.051507 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:39.051415 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:39:39.347853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:39.347826 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4"] Sep 10 19:39:39.350437 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:39:39.350409 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87f0afe_5140_4636_a690_bbc13cf75d78.slice/crio-6810a903c03f558bf8c3ea20bfc39874a2c7188202337a7d15f1f57ebfc8174b WatchSource:0}: Error finding container 6810a903c03f558bf8c3ea20bfc39874a2c7188202337a7d15f1f57ebfc8174b: Status 404 returned error can't find the container with id 6810a903c03f558bf8c3ea20bfc39874a2c7188202337a7d15f1f57ebfc8174b Sep 10 19:39:39.904339 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:39.904296 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" event={"ID":"f87f0afe-5140-4636-a690-bbc13cf75d78","Type":"ContainerStarted","Data":"2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8"} Sep 10 19:39:39.904736 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:39.904376 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" event={"ID":"f87f0afe-5140-4636-a690-bbc13cf75d78","Type":"ContainerStarted","Data":"6810a903c03f558bf8c3ea20bfc39874a2c7188202337a7d15f1f57ebfc8174b"} Sep 10 19:39:39.904736 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:39.904398 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:39:39.922350 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:39.922274 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" podStartSLOduration=1.9222566730000001 podStartE2EDuration="1.922256673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:39:39.92014352 +0000 UTC m=+1165.887955768" watchObservedRunningTime="2026-09-10 19:39:39.922256673 +0000 UTC m=+1165.890068919" Sep 10 19:39:50.912084 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:50.912050 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:39:51.920515 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:51.920468 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 19:39:51.920930 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:51.920556 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 19:39:52.978626 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:52.978521 2584 generic.go:356] "Generic (PLEG): container finished" podID="cfac205e-0cb1-426c-8411-1dad1d224947" containerID="f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7" exitCode=0 Sep 10 19:39:52.978626 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:52.978591 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" event={"ID":"cfac205e-0cb1-426c-8411-1dad1d224947","Type":"ContainerDied","Data":"f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7"} Sep 10 19:39:52.979312 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:52.978999 2584 scope.go:117] "RemoveContainer" containerID="f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7" Sep 10 19:39:52.980047 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:52.980021 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:52.980108 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:52.980091 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:52.980150 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:52.980122 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:39:53.985085 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:53.985048 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" event={"ID":"cfac205e-0cb1-426c-8411-1dad1d224947","Type":"ContainerStarted","Data":"cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7"} Sep 10 19:39:54.499347 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:54.499310 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:54.499583 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:54.499564 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 19:39:54.499631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:54.499607 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 19:39:55.498982 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:55.498936 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 19:39:55.499431 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:55.499002 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 19:39:56.498927 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:56.498891 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 19:39:56.499113 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:56.498947 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 19:39:57.499412 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:57.499379 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 19:39:57.499798 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:57.499436 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 19:39:58.498953 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:58.498913 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 19:39:58.499139 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:58.498973 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 19:39:59.712617 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:59.712579 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:59.713112 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:59.713082 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:39:59.714026 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:39:59.714005 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:40:03.040198 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.040161 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw"] Sep 10 19:40:03.042794 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.042774 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:40:03.045032 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.045006 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ecadb8\"" Sep 10 19:40:03.045175 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.045030 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ecadb8-dockercfg-h5n29\"" Sep 10 19:40:03.060885 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.060848 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw"] Sep 10 19:40:03.106991 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.106939 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1df488-399f-4626-b620-8d2ae77848e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw\" (UID: \"db1df488-399f-4626-b620-8d2ae77848e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:40:03.107176 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.107006 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4xw\" (UniqueName: \"kubernetes.io/projected/db1df488-399f-4626-b620-8d2ae77848e2-kube-api-access-lx4xw\") pod \"payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw\" (UID: \"db1df488-399f-4626-b620-8d2ae77848e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:40:03.208435 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.208385 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1df488-399f-4626-b620-8d2ae77848e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw\" (UID: \"db1df488-399f-4626-b620-8d2ae77848e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:40:03.208628 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.208451 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4xw\" (UniqueName: \"kubernetes.io/projected/db1df488-399f-4626-b620-8d2ae77848e2-kube-api-access-lx4xw\") pod \"payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw\" (UID: \"db1df488-399f-4626-b620-8d2ae77848e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:40:03.209125 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.209102 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1df488-399f-4626-b620-8d2ae77848e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw\" (UID: \"db1df488-399f-4626-b620-8d2ae77848e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:40:03.217259 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.217227 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4xw\" (UniqueName: \"kubernetes.io/projected/db1df488-399f-4626-b620-8d2ae77848e2-kube-api-access-lx4xw\") pod \"payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw\" (UID: \"db1df488-399f-4626-b620-8d2ae77848e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:40:03.353765 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.353727 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:40:03.626525 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:03.626494 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw"] Sep 10 19:40:03.628763 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:40:03.628733 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1df488_399f_4626_b620_8d2ae77848e2.slice/crio-3a422b9b2d79ff2b07e42b7b20cafc32c9498cdb8ef3793b57ec666b3fbfde63 WatchSource:0}: Error finding container 3a422b9b2d79ff2b07e42b7b20cafc32c9498cdb8ef3793b57ec666b3fbfde63: Status 404 returned error can't find the container with id 3a422b9b2d79ff2b07e42b7b20cafc32c9498cdb8ef3793b57ec666b3fbfde63 Sep 10 19:40:04.036326 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:04.036217 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" event={"ID":"db1df488-399f-4626-b620-8d2ae77848e2","Type":"ContainerStarted","Data":"3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3"} Sep 10 19:40:04.036326 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:04.036296 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" event={"ID":"db1df488-399f-4626-b620-8d2ae77848e2","Type":"ContainerStarted","Data":"3a422b9b2d79ff2b07e42b7b20cafc32c9498cdb8ef3793b57ec666b3fbfde63"} Sep 10 19:40:04.036521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:04.036328 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:40:04.052053 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:04.051992 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" podStartSLOduration=1.051971865 podStartE2EDuration="1.051971865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:40:04.05151393 +0000 UTC m=+1190.019326178" watchObservedRunningTime="2026-09-10 19:40:04.051971865 +0000 UTC m=+1190.019784114" Sep 10 19:40:15.044546 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:15.044457 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:40:26.433786 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.433750 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb"] Sep 10 19:40:26.438729 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.438700 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.441022 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.440993 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ff3019-openshift-default-dockercfg-vpg9j\"" Sep 10 19:40:26.450989 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.450961 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb"] Sep 10 19:40:26.527059 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.527015 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-certs\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.527059 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.527066 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-socket\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.527339 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.527093 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttdh\" (UniqueName: \"kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-kube-api-access-zttdh\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.527339 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.527146 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istiod-ca-cert\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.527339 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.527195 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-envoy\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.527339 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.527216 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-token\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.527339 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.527252 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-data\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.527339 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.527328 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-credential-socket\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.527567 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.527381 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-podinfo\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.628653 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.628612 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-podinfo\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.628891 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.628708 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-certs\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.628891 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.628770 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-socket\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.628891 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.628846 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttdh\" (UniqueName: \"kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-kube-api-access-zttdh\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.628891 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.628879 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istiod-ca-cert\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.629127 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.628968 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-envoy\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.629127 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.629000 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-token\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.629127 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.629055 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-data\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.629301 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.629132 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-credential-socket\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.629301 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.629217 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-socket\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.629301 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.629244 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-certs\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.629734 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.629709 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-credential-socket\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.629844 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.629712 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-data\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.629911 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.629881 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istiod-ca-cert\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.631361 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.631342 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-podinfo\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.631613 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.631592 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-envoy\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.638110 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.638080 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-token\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.638194 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.638179 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttdh\" (UniqueName: \"kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-kube-api-access-zttdh\") pod \"e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:26.750734 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:26.750654 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:27.028581 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:27.028554 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb"] Sep 10 19:40:27.030874 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:40:27.030846 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1bb938_a6d4_42c7_8771_a82c6bcf536d.slice/crio-6747296fc9e68bf0d5158b633dc67238bb26e6fff66dcc7ac94e5e50b50a21ad WatchSource:0}: Error finding container 6747296fc9e68bf0d5158b633dc67238bb26e6fff66dcc7ac94e5e50b50a21ad: Status 404 returned error can't find the container with id 6747296fc9e68bf0d5158b633dc67238bb26e6fff66dcc7ac94e5e50b50a21ad Sep 10 19:40:27.033621 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:27.033589 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:40:27.033723 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:27.033660 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:40:27.033723 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:27.033688 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:40:27.156650 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:27.156603 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" event={"ID":"ba1bb938-a6d4-42c7-8771-a82c6bcf536d","Type":"ContainerStarted","Data":"5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d"} Sep 10 19:40:27.156868 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:27.156669 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" event={"ID":"ba1bb938-a6d4-42c7-8771-a82c6bcf536d","Type":"ContainerStarted","Data":"6747296fc9e68bf0d5158b633dc67238bb26e6fff66dcc7ac94e5e50b50a21ad"} Sep 10 19:40:27.183365 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:27.177336 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" podStartSLOduration=1.177312995 podStartE2EDuration="1.177312995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:40:27.174935627 +0000 UTC m=+1213.142747865" watchObservedRunningTime="2026-09-10 19:40:27.177312995 +0000 UTC m=+1213.145125242" Sep 10 19:40:27.751021 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:27.750979 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:28.757130 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:28.757093 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:29.167492 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:29.167462 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:29.168405 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:29.168384 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:30.931347 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:30.931307 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s"] Sep 10 19:40:30.935045 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:30.935025 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:40:30.937498 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:30.937472 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ff3019\"" Sep 10 19:40:30.946151 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:30.946124 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s"] Sep 10 19:40:30.967587 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:30.967544 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2"] Sep 10 19:40:30.971623 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:30.971599 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:40:30.973916 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:30.973892 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ff3019-dockercfg-dcsl8\"" Sep 10 19:40:30.980401 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:30.980378 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2"] Sep 10 19:40:31.071706 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.071662 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swb9l\" (UniqueName: \"kubernetes.io/projected/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-kube-api-access-swb9l\") pod \"payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2\" (UID: \"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:40:31.072039 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.072014 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5fcp\" (UniqueName: \"kubernetes.io/projected/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-kube-api-access-c5fcp\") pod \"payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s\" (UID: \"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:40:31.072179 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.072164 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2\" (UID: \"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:40:31.072358 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.072340 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s\" (UID: \"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:40:31.173778 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.173741 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s\" (UID: \"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:40:31.173983 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.173823 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swb9l\" (UniqueName: \"kubernetes.io/projected/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-kube-api-access-swb9l\") pod \"payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2\" (UID: \"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:40:31.173983 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.173899 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5fcp\" (UniqueName: \"kubernetes.io/projected/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-kube-api-access-c5fcp\") pod \"payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s\" (UID: \"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:40:31.173983 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.173942 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2\" (UID: \"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:40:31.174706 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.174679 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2\" (UID: \"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:40:31.174808 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.174783 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s\" (UID: \"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:40:31.182146 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.182075 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5fcp\" (UniqueName: \"kubernetes.io/projected/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-kube-api-access-c5fcp\") pod \"payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s\" (UID: \"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:40:31.182604 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.182581 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swb9l\" (UniqueName: \"kubernetes.io/projected/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-kube-api-access-swb9l\") pod \"payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2\" (UID: \"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:40:31.247751 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.247700 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:40:31.284583 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.284550 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:40:31.535316 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.535273 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s"] Sep 10 19:40:31.536596 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:40:31.536559 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a2f13c_4814_4ab1_9046_fa84bb1ca5ca.slice/crio-8b08006cba4082d3464af2af1a03aee10d7f67b40fc8ed78be7edadfe7fb4ade WatchSource:0}: Error finding container 8b08006cba4082d3464af2af1a03aee10d7f67b40fc8ed78be7edadfe7fb4ade: Status 404 returned error can't find the container with id 8b08006cba4082d3464af2af1a03aee10d7f67b40fc8ed78be7edadfe7fb4ade Sep 10 19:40:31.570981 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:31.570953 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2"] Sep 10 19:40:31.572830 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:40:31.572804 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2632965c_2476_4d1a_96ef_4f8dc3cfb8ae.slice/crio-72c71fcfe2f9a23755deeca3003a74465b32fb4e5e51c88563801934a185c21e WatchSource:0}: Error finding container 72c71fcfe2f9a23755deeca3003a74465b32fb4e5e51c88563801934a185c21e: Status 404 returned error can't find the container with id 72c71fcfe2f9a23755deeca3003a74465b32fb4e5e51c88563801934a185c21e Sep 10 19:40:32.187058 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:32.187021 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" event={"ID":"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae","Type":"ContainerStarted","Data":"2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6"} Sep 10 19:40:32.187558 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:32.187079 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" event={"ID":"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae","Type":"ContainerStarted","Data":"72c71fcfe2f9a23755deeca3003a74465b32fb4e5e51c88563801934a185c21e"} Sep 10 19:40:32.187558 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:32.187108 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:40:32.188919 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:32.188894 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" event={"ID":"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca","Type":"ContainerStarted","Data":"029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e"} Sep 10 19:40:32.189069 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:32.188939 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" event={"ID":"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca","Type":"ContainerStarted","Data":"8b08006cba4082d3464af2af1a03aee10d7f67b40fc8ed78be7edadfe7fb4ade"} Sep 10 19:40:32.189069 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:32.188975 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:40:32.205936 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:32.205871 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" podStartSLOduration=2.205850299 podStartE2EDuration="2.205850299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:40:32.203499151 +0000 UTC m=+1218.171311421" watchObservedRunningTime="2026-09-10 19:40:32.205850299 +0000 UTC m=+1218.173662549" Sep 10 19:40:32.220116 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:32.220057 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" podStartSLOduration=2.220034028 podStartE2EDuration="2.220034028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:40:32.218477401 +0000 UTC m=+1218.186289652" watchObservedRunningTime="2026-09-10 19:40:32.220034028 +0000 UTC m=+1218.187846278" Sep 10 19:40:43.196840 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:43.196808 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:40:43.197243 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:43.196884 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:40:45.209496 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:45.209443 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 19:40:45.209936 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:45.209568 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 19:40:47.303670 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:47.303638 2584 generic.go:356] "Generic (PLEG): container finished" podID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerID="5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d" exitCode=0 Sep 10 19:40:47.304077 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:47.303694 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" event={"ID":"ba1bb938-a6d4-42c7-8771-a82c6bcf536d","Type":"ContainerDied","Data":"5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d"} Sep 10 19:40:47.304121 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:47.304074 2584 scope.go:117] "RemoveContainer" containerID="5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d" Sep 10 19:40:47.305233 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:47.305198 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:40:47.305296 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:47.305287 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:40:47.305337 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:47.305319 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:40:48.311737 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:48.311701 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" event={"ID":"ba1bb938-a6d4-42c7-8771-a82c6bcf536d","Type":"ContainerStarted","Data":"61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2"} Sep 10 19:40:48.751794 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:48.751694 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:48.751958 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:48.751937 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 19:40:48.752010 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:48.751987 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 19:40:49.751853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:49.751811 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 19:40:49.752357 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:49.751897 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 19:40:50.751908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:50.751871 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 19:40:50.752343 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:50.751932 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 19:40:51.751592 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:51.751552 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 19:40:51.751781 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:51.751613 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 19:40:52.751314 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:52.751248 2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 19:40:52.751714 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:52.751344 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 19:40:53.755459 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:53.755417 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:53.755874 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:53.755847 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:40:53.756469 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:40:53.756450 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:41:02.477427 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.477387 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59"] Sep 10 19:41:02.480294 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.480261 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.483932 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.483901 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1fa9\"/\"default-dockercfg-ghwlj\"" Sep 10 19:41:02.483932 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.483919 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1fa9\"/\"openshift-service-ca.crt\"" Sep 10 19:41:02.484147 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.483929 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1fa9\"/\"auth-test-model-2a1fa9-kserve-self-signed-certs\"" Sep 10 19:41:02.484147 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.483912 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1fa9\"/\"kube-root-ca.crt\"" Sep 10 19:41:02.491269 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.491239 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59"] Sep 10 19:41:02.581810 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.581768 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5435d4-43b9-47ae-959c-2c8332dcecae-tls-certs\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.582021 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.581820 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-model-cache\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.582021 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.581846 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-home\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.582021 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.581890 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-tmp-dir\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.582021 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.581947 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-dshm\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.582021 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.581988 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qq9f\" (UniqueName: \"kubernetes.io/projected/ca5435d4-43b9-47ae-959c-2c8332dcecae-kube-api-access-4qq9f\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.683233 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.683191 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-model-cache\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.683233 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.683239 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-home\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.683546 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.683272 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-tmp-dir\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.683546 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.683336 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-dshm\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.683546 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.683369 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qq9f\" (UniqueName: \"kubernetes.io/projected/ca5435d4-43b9-47ae-959c-2c8332dcecae-kube-api-access-4qq9f\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.683546 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.683462 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5435d4-43b9-47ae-959c-2c8332dcecae-tls-certs\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.683759 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.683711 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-tmp-dir\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.683818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.683773 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-model-cache\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.683859 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.683818 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-home\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.685791 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.685772 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-dshm\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.685902 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.685884 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5435d4-43b9-47ae-959c-2c8332dcecae-tls-certs\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.691551 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.691521 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qq9f\" (UniqueName: \"kubernetes.io/projected/ca5435d4-43b9-47ae-959c-2c8332dcecae-kube-api-access-4qq9f\") pod \"auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:02.790910 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:02.790809 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:03.068255 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:03.068227 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59"] Sep 10 19:41:03.070422 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:41:03.070391 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca5435d4_43b9_47ae_959c_2c8332dcecae.slice/crio-c9589706f73e59b78bce5394499aed100a1b52b1f91aa65a2d385e757cc84a8a WatchSource:0}: Error finding container c9589706f73e59b78bce5394499aed100a1b52b1f91aa65a2d385e757cc84a8a: Status 404 returned error can't find the container with id c9589706f73e59b78bce5394499aed100a1b52b1f91aa65a2d385e757cc84a8a Sep 10 19:41:03.403999 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:03.403913 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" event={"ID":"ca5435d4-43b9-47ae-959c-2c8332dcecae","Type":"ContainerStarted","Data":"4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1"} Sep 10 19:41:03.403999 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:03.403957 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" event={"ID":"ca5435d4-43b9-47ae-959c-2c8332dcecae","Type":"ContainerStarted","Data":"c9589706f73e59b78bce5394499aed100a1b52b1f91aa65a2d385e757cc84a8a"} Sep 10 19:41:03.420929 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:03.420871 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" podStartSLOduration=1.420854194 podStartE2EDuration="1.420854194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:41:03.420221528 +0000 UTC m=+1249.388033789" watchObservedRunningTime="2026-09-10 19:41:03.420854194 +0000 UTC m=+1249.388666441" Sep 10 19:41:12.791919 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:12.791824 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:12.791919 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:12.791875 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:12.804581 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:12.804554 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:13.467400 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:13.467366 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:41:14.573163 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:41:14.573129 2584 scope.go:117] "RemoveContainer" containerID="e38382bfdc4cebf54b2889c6401555831abd38c92172bfacc440068f0381e44a" Sep 10 19:43:14.600806 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:14.600771 2584 scope.go:117] "RemoveContainer" containerID="eb144481452dcba9c79169e74defca91dd54624a265bd885fa51ecd91729baa2" Sep 10 19:43:16.079252 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:16.079201 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59"] Sep 10 19:43:31.104845 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.104793 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" podUID="ca5435d4-43b9-47ae-959c-2c8332dcecae" containerName="main" containerID="cri-o://4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1" gracePeriod=45 Sep 10 19:43:31.354190 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.354165 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:43:31.496807 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.496690 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-model-cache\") pod \"ca5435d4-43b9-47ae-959c-2c8332dcecae\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " Sep 10 19:43:31.496807 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.496804 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-home\") pod \"ca5435d4-43b9-47ae-959c-2c8332dcecae\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " Sep 10 19:43:31.497042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.496864 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-tmp-dir\") pod \"ca5435d4-43b9-47ae-959c-2c8332dcecae\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " Sep 10 19:43:31.497042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.496898 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-dshm\") pod \"ca5435d4-43b9-47ae-959c-2c8332dcecae\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " Sep 10 19:43:31.497042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.496968 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5435d4-43b9-47ae-959c-2c8332dcecae-tls-certs\") pod \"ca5435d4-43b9-47ae-959c-2c8332dcecae\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " Sep 10 19:43:31.497042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.496998 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qq9f\" (UniqueName: \"kubernetes.io/projected/ca5435d4-43b9-47ae-959c-2c8332dcecae-kube-api-access-4qq9f\") pod \"ca5435d4-43b9-47ae-959c-2c8332dcecae\" (UID: \"ca5435d4-43b9-47ae-959c-2c8332dcecae\") " Sep 10 19:43:31.497042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.497019 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca5435d4-43b9-47ae-959c-2c8332dcecae" (UID: "ca5435d4-43b9-47ae-959c-2c8332dcecae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:43:31.497265 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.497039 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-home" (OuterVolumeSpecName: "home") pod "ca5435d4-43b9-47ae-959c-2c8332dcecae" (UID: "ca5435d4-43b9-47ae-959c-2c8332dcecae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:43:31.497265 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.497104 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca5435d4-43b9-47ae-959c-2c8332dcecae" (UID: "ca5435d4-43b9-47ae-959c-2c8332dcecae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:43:31.497265 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.497251 2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-model-cache\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:43:31.497443 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.497269 2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-home\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:43:31.497443 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.497313 2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-tmp-dir\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:43:31.499347 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.499317 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-dshm" (OuterVolumeSpecName: "dshm") pod "ca5435d4-43b9-47ae-959c-2c8332dcecae" (UID: "ca5435d4-43b9-47ae-959c-2c8332dcecae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:43:31.499465 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.499381 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5435d4-43b9-47ae-959c-2c8332dcecae-kube-api-access-4qq9f" (OuterVolumeSpecName: "kube-api-access-4qq9f") pod "ca5435d4-43b9-47ae-959c-2c8332dcecae" (UID: "ca5435d4-43b9-47ae-959c-2c8332dcecae"). InnerVolumeSpecName "kube-api-access-4qq9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:43:31.499465 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.499391 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5435d4-43b9-47ae-959c-2c8332dcecae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca5435d4-43b9-47ae-959c-2c8332dcecae" (UID: "ca5435d4-43b9-47ae-959c-2c8332dcecae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:43:31.598273 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.598233 2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5435d4-43b9-47ae-959c-2c8332dcecae-dshm\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:43:31.598273 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.598264 2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5435d4-43b9-47ae-959c-2c8332dcecae-tls-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:43:31.598273 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:31.598288 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qq9f\" (UniqueName: \"kubernetes.io/projected/ca5435d4-43b9-47ae-959c-2c8332dcecae-kube-api-access-4qq9f\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:43:32.183777 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:32.183739 2584 generic.go:356] "Generic (PLEG): container finished" podID="ca5435d4-43b9-47ae-959c-2c8332dcecae" containerID="4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1" exitCode=0 Sep 10 19:43:32.184257 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:32.183789 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" Sep 10 19:43:32.184257 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:32.183826 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" event={"ID":"ca5435d4-43b9-47ae-959c-2c8332dcecae","Type":"ContainerDied","Data":"4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1"} Sep 10 19:43:32.184257 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:32.183879 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59" event={"ID":"ca5435d4-43b9-47ae-959c-2c8332dcecae","Type":"ContainerDied","Data":"c9589706f73e59b78bce5394499aed100a1b52b1f91aa65a2d385e757cc84a8a"} Sep 10 19:43:32.184257 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:32.183906 2584 scope.go:117] "RemoveContainer" containerID="4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1" Sep 10 19:43:32.193833 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:32.193815 2584 scope.go:117] "RemoveContainer" containerID="4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1" Sep 10 19:43:32.194157 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:43:32.194135 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1\": container with ID starting with 4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1 not found: ID does not exist" containerID="4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1" Sep 10 19:43:32.194207 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:32.194167 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1"} err="failed to get container status \"4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1\": rpc error: code = NotFound desc = could not find container \"4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1\": container with ID starting with 4e22390c433dbb3c99b0294ba3e83ac8871c7c1e86c4a5579236649882befec1 not found: ID does not exist" Sep 10 19:43:32.211099 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:32.211064 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59"] Sep 10 19:43:32.212985 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:32.212957 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/auth-test-model-2a1fa9-kserve-7bf57d5bcf-p5q59"] Sep 10 19:43:32.489053 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:43:32.488951 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca5435d4-43b9-47ae-959c-2c8332dcecae" path="/var/lib/kubelet/pods/ca5435d4-43b9-47ae-959c-2c8332dcecae/volumes" Sep 10 19:44:02.183272 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.183228 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr"] Sep 10 19:44:02.183986 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.183961 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5435d4-43b9-47ae-959c-2c8332dcecae" containerName="main" Sep 10 19:44:02.184058 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.183990 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5435d4-43b9-47ae-959c-2c8332dcecae" containerName="main" Sep 10 19:44:02.184122 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.184108 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5435d4-43b9-47ae-959c-2c8332dcecae" containerName="main" Sep 10 19:44:02.187237 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.187215 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.190749 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.190720 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1fa9\"/\"openshift-service-ca.crt\"" Sep 10 19:44:02.190889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.190720 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1fa9\"/\"sub-test-model-2a1fa9-kserve-self-signed-certs\"" Sep 10 19:44:02.190889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.190720 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1fa9\"/\"kube-root-ca.crt\"" Sep 10 19:44:02.190889 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.190723 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1fa9\"/\"default-dockercfg-ghwlj\"" Sep 10 19:44:02.197175 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.197145 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr"] Sep 10 19:44:02.303212 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.303167 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-tmp-dir\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.303435 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.303220 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-dshm\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.303435 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.303243 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-model-cache\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.303435 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.303359 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99061b63-f117-4aa3-914f-99de52674420-tls-certs\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.303435 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.303389 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-home\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.303586 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.303443 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsj2n\" (UniqueName: \"kubernetes.io/projected/99061b63-f117-4aa3-914f-99de52674420-kube-api-access-zsj2n\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.404304 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.404224 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-tmp-dir\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.404523 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.404334 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-dshm\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.404523 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.404370 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-model-cache\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.404523 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.404403 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99061b63-f117-4aa3-914f-99de52674420-tls-certs\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.404523 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.404436 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-home\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.404523 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.404477 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsj2n\" (UniqueName: \"kubernetes.io/projected/99061b63-f117-4aa3-914f-99de52674420-kube-api-access-zsj2n\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.404813 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.404749 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-tmp-dir\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.405063 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.405042 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-model-cache\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.405469 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.405417 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-home\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.407914 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.407885 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99061b63-f117-4aa3-914f-99de52674420-tls-certs\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.408039 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.407932 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-dshm\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.415067 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.414999 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsj2n\" (UniqueName: \"kubernetes.io/projected/99061b63-f117-4aa3-914f-99de52674420-kube-api-access-zsj2n\") pod \"sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.498324 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.498206 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:02.796219 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:02.796181 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr"] Sep 10 19:44:02.798268 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:44:02.798238 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99061b63_f117_4aa3_914f_99de52674420.slice/crio-39da3bccb5aba89c32fa362895440eef5d7d2eb588d9c3d2179c61a76571c3c3 WatchSource:0}: Error finding container 39da3bccb5aba89c32fa362895440eef5d7d2eb588d9c3d2179c61a76571c3c3: Status 404 returned error can't find the container with id 39da3bccb5aba89c32fa362895440eef5d7d2eb588d9c3d2179c61a76571c3c3 Sep 10 19:44:03.357355 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:03.357315 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" event={"ID":"99061b63-f117-4aa3-914f-99de52674420","Type":"ContainerStarted","Data":"1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f"} Sep 10 19:44:03.357355 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:03.357364 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" event={"ID":"99061b63-f117-4aa3-914f-99de52674420","Type":"ContainerStarted","Data":"39da3bccb5aba89c32fa362895440eef5d7d2eb588d9c3d2179c61a76571c3c3"} Sep 10 19:44:03.387726 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:03.387661 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" podStartSLOduration=1.387638442 podStartE2EDuration="1.387638442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:44:03.385499191 +0000 UTC m=+1429.353311440" watchObservedRunningTime="2026-09-10 19:44:03.387638442 +0000 UTC m=+1429.355450691" Sep 10 19:44:12.498736 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:12.498645 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:12.498736 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:12.498704 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:12.511901 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:12.511869 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:13.421130 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:13.421092 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:44:14.622143 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:14.622106 2584 scope.go:117] "RemoveContainer" containerID="77538296e428538cebdb2a67e45d9da3cb95aa8a6da62fb66567284f2a5c8f36" Sep 10 19:44:55.970035 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:44:55.969998 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr"] Sep 10 19:45:00.177571 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.177530 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q"] Sep 10 19:45:00.181412 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.181387 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q" Sep 10 19:45:00.182063 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.182040 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817825-nw9ms"] Sep 10 19:45:00.184404 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.184382 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zztzd\"" Sep 10 19:45:00.185475 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.185458 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-nw9ms" Sep 10 19:45:00.192896 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.192858 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q"] Sep 10 19:45:00.194787 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.194761 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q" Sep 10 19:45:00.203793 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.203755 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817825-nw9ms"] Sep 10 19:45:00.205441 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.205418 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-nw9ms" Sep 10 19:45:00.210676 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.210643 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89"] Sep 10 19:45:00.215663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.215638 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89" Sep 10 19:45:00.226247 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.226163 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89"] Sep 10 19:45:00.252818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.252789 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89" Sep 10 19:45:00.491549 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.491521 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q"] Sep 10 19:45:00.493384 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:45:00.493342 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d36e27_57a1_492e_9cc4_8da9c0f78b8e.slice/crio-86f0cf0533aad5dc49b445dd8526ad7223f3543812acde196846849a70808752 WatchSource:0}: Error finding container 86f0cf0533aad5dc49b445dd8526ad7223f3543812acde196846849a70808752: Status 404 returned error can't find the container with id 86f0cf0533aad5dc49b445dd8526ad7223f3543812acde196846849a70808752 Sep 10 19:45:00.499486 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.499462 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817825-nw9ms"] Sep 10 19:45:00.501972 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:45:00.501945 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d9faa3_0e8f_4225_987d_4548a8970ebd.slice/crio-045de414fd52f258b5558f5f27098705d3f14aeb7da358fbb73841cf56ee5479 WatchSource:0}: Error finding container 045de414fd52f258b5558f5f27098705d3f14aeb7da358fbb73841cf56ee5479: Status 404 returned error can't find the container with id 045de414fd52f258b5558f5f27098705d3f14aeb7da358fbb73841cf56ee5479 Sep 10 19:45:00.558251 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.558220 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89"] Sep 10 19:45:00.560314 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:45:00.560263 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1260fb_9ca8_499c_90f4_b7630ed735ec.slice/crio-98a8246be434c4bcd8ce83c474e3daac9a56478a0869d885edccbe4359b47c73 WatchSource:0}: Error finding container 98a8246be434c4bcd8ce83c474e3daac9a56478a0869d885edccbe4359b47c73: Status 404 returned error can't find the container with id 98a8246be434c4bcd8ce83c474e3daac9a56478a0869d885edccbe4359b47c73 Sep 10 19:45:00.655090 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.655052 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-nw9ms" event={"ID":"04d9faa3-0e8f-4225-987d-4548a8970ebd","Type":"ContainerStarted","Data":"045de414fd52f258b5558f5f27098705d3f14aeb7da358fbb73841cf56ee5479"} Sep 10 19:45:00.656228 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.656194 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89" event={"ID":"ee1260fb-9ca8-499c-90f4-b7630ed735ec","Type":"ContainerStarted","Data":"98a8246be434c4bcd8ce83c474e3daac9a56478a0869d885edccbe4359b47c73"} Sep 10 19:45:00.657391 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:00.657365 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q" event={"ID":"c3d36e27-57a1-492e-9cc4-8da9c0f78b8e","Type":"ContainerStarted","Data":"86f0cf0533aad5dc49b445dd8526ad7223f3543812acde196846849a70808752"} Sep 10 19:45:01.664362 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:01.664328 2584 generic.go:356] "Generic (PLEG): container finished" podID="04d9faa3-0e8f-4225-987d-4548a8970ebd" containerID="79399188948f576b7ee448a9dabcb89884b01cb37e0423ae802d2b9049f9fd0d" exitCode=0 Sep 10 19:45:01.664833 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:01.664428 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-nw9ms" event={"ID":"04d9faa3-0e8f-4225-987d-4548a8970ebd","Type":"ContainerDied","Data":"79399188948f576b7ee448a9dabcb89884b01cb37e0423ae802d2b9049f9fd0d"} Sep 10 19:45:01.665966 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:01.665943 2584 generic.go:356] "Generic (PLEG): container finished" podID="ee1260fb-9ca8-499c-90f4-b7630ed735ec" containerID="054ea0e2689f3adbdc7508d79f92ece8f46f64281965a968316701bc40f9c9cf" exitCode=0 Sep 10 19:45:01.666054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:01.666008 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89" event={"ID":"ee1260fb-9ca8-499c-90f4-b7630ed735ec","Type":"ContainerDied","Data":"054ea0e2689f3adbdc7508d79f92ece8f46f64281965a968316701bc40f9c9cf"} Sep 10 19:45:01.667480 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:01.667453 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q" event={"ID":"c3d36e27-57a1-492e-9cc4-8da9c0f78b8e","Type":"ContainerStarted","Data":"cfea5e6964fe7ec76123e3cf28e50c0828e889d061167bfda16c1ac3a0079806"} Sep 10 19:45:01.717375 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:01.717319 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q" podStartSLOduration=1.7172741500000002 podStartE2EDuration="1.71727415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:45:01.71505198 +0000 UTC m=+1487.682864227" watchObservedRunningTime="2026-09-10 19:45:01.71727415 +0000 UTC m=+1487.685086398" Sep 10 19:45:02.680355 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:02.680313 2584 generic.go:356] "Generic (PLEG): container finished" podID="c3d36e27-57a1-492e-9cc4-8da9c0f78b8e" containerID="cfea5e6964fe7ec76123e3cf28e50c0828e889d061167bfda16c1ac3a0079806" exitCode=0 Sep 10 19:45:02.680802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:02.680394 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q" event={"ID":"c3d36e27-57a1-492e-9cc4-8da9c0f78b8e","Type":"ContainerDied","Data":"cfea5e6964fe7ec76123e3cf28e50c0828e889d061167bfda16c1ac3a0079806"} Sep 10 19:45:02.863996 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:02.863967 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89" Sep 10 19:45:02.867036 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:02.867012 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-nw9ms" Sep 10 19:45:03.689603 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:03.689572 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-nw9ms" Sep 10 19:45:03.690010 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:03.689576 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-nw9ms" event={"ID":"04d9faa3-0e8f-4225-987d-4548a8970ebd","Type":"ContainerDied","Data":"045de414fd52f258b5558f5f27098705d3f14aeb7da358fbb73841cf56ee5479"} Sep 10 19:45:03.690010 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:03.689708 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="045de414fd52f258b5558f5f27098705d3f14aeb7da358fbb73841cf56ee5479" Sep 10 19:45:03.691096 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:03.691065 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89" Sep 10 19:45:03.691202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:03.691096 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89" event={"ID":"ee1260fb-9ca8-499c-90f4-b7630ed735ec","Type":"ContainerDied","Data":"98a8246be434c4bcd8ce83c474e3daac9a56478a0869d885edccbe4359b47c73"} Sep 10 19:45:03.691202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:03.691139 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98a8246be434c4bcd8ce83c474e3daac9a56478a0869d885edccbe4359b47c73" Sep 10 19:45:03.838724 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:03.838690 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q" Sep 10 19:45:04.698940 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:04.698904 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q" event={"ID":"c3d36e27-57a1-492e-9cc4-8da9c0f78b8e","Type":"ContainerDied","Data":"86f0cf0533aad5dc49b445dd8526ad7223f3543812acde196846849a70808752"} Sep 10 19:45:04.699328 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:04.698959 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86f0cf0533aad5dc49b445dd8526ad7223f3543812acde196846849a70808752" Sep 10 19:45:04.699328 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:04.698914 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q" Sep 10 19:45:10.990109 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:10.990064 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" podUID="99061b63-f117-4aa3-914f-99de52674420" containerName="main" containerID="cri-o://1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f" gracePeriod=45 Sep 10 19:45:11.253996 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.253929 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:45:11.301494 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.301459 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-home\") pod \"99061b63-f117-4aa3-914f-99de52674420\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " Sep 10 19:45:11.301698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.301547 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsj2n\" (UniqueName: \"kubernetes.io/projected/99061b63-f117-4aa3-914f-99de52674420-kube-api-access-zsj2n\") pod \"99061b63-f117-4aa3-914f-99de52674420\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " Sep 10 19:45:11.301698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.301596 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-model-cache\") pod \"99061b63-f117-4aa3-914f-99de52674420\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " Sep 10 19:45:11.301698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.301633 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-tmp-dir\") pod \"99061b63-f117-4aa3-914f-99de52674420\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " Sep 10 19:45:11.301698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.301663 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99061b63-f117-4aa3-914f-99de52674420-tls-certs\") pod \"99061b63-f117-4aa3-914f-99de52674420\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " Sep 10 19:45:11.301698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.301686 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-dshm\") pod \"99061b63-f117-4aa3-914f-99de52674420\" (UID: \"99061b63-f117-4aa3-914f-99de52674420\") " Sep 10 19:45:11.301996 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.301752 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-home" (OuterVolumeSpecName: "home") pod "99061b63-f117-4aa3-914f-99de52674420" (UID: "99061b63-f117-4aa3-914f-99de52674420"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:45:11.301996 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.301927 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99061b63-f117-4aa3-914f-99de52674420" (UID: "99061b63-f117-4aa3-914f-99de52674420"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:45:11.301996 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.301943 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-model-cache" (OuterVolumeSpecName: "model-cache") pod "99061b63-f117-4aa3-914f-99de52674420" (UID: "99061b63-f117-4aa3-914f-99de52674420"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:45:11.302164 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.302058 2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-model-cache\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:45:11.302164 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.302080 2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-tmp-dir\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:45:11.302164 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.302095 2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-home\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:45:11.303802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.303775 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-dshm" (OuterVolumeSpecName: "dshm") pod "99061b63-f117-4aa3-914f-99de52674420" (UID: "99061b63-f117-4aa3-914f-99de52674420"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:45:11.303802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.303799 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99061b63-f117-4aa3-914f-99de52674420-kube-api-access-zsj2n" (OuterVolumeSpecName: "kube-api-access-zsj2n") pod "99061b63-f117-4aa3-914f-99de52674420" (UID: "99061b63-f117-4aa3-914f-99de52674420"). InnerVolumeSpecName "kube-api-access-zsj2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:45:11.303983 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.303856 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99061b63-f117-4aa3-914f-99de52674420-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99061b63-f117-4aa3-914f-99de52674420" (UID: "99061b63-f117-4aa3-914f-99de52674420"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:45:11.402873 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.402827 2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99061b63-f117-4aa3-914f-99de52674420-tls-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:45:11.402873 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.402865 2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99061b63-f117-4aa3-914f-99de52674420-dshm\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:45:11.402873 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.402874 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsj2n\" (UniqueName: \"kubernetes.io/projected/99061b63-f117-4aa3-914f-99de52674420-kube-api-access-zsj2n\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:45:11.735722 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.735684 2584 generic.go:356] "Generic (PLEG): container finished" podID="99061b63-f117-4aa3-914f-99de52674420" containerID="1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f" exitCode=0 Sep 10 19:45:11.735909 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.735728 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" Sep 10 19:45:11.735909 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.735764 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" event={"ID":"99061b63-f117-4aa3-914f-99de52674420","Type":"ContainerDied","Data":"1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f"} Sep 10 19:45:11.735909 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.735809 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr" event={"ID":"99061b63-f117-4aa3-914f-99de52674420","Type":"ContainerDied","Data":"39da3bccb5aba89c32fa362895440eef5d7d2eb588d9c3d2179c61a76571c3c3"} Sep 10 19:45:11.735909 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.735826 2584 scope.go:117] "RemoveContainer" containerID="1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f" Sep 10 19:45:11.744914 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.744893 2584 scope.go:117] "RemoveContainer" containerID="1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f" Sep 10 19:45:11.745235 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:45:11.745217 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f\": container with ID starting with 1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f not found: ID does not exist" containerID="1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f" Sep 10 19:45:11.745308 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.745245 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f"} err="failed to get container status \"1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f\": rpc error: code = NotFound desc = could not find container \"1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f\": container with ID starting with 1cfc56987e61f2ccfd72a7eb1f446fb2fb62f2d042883b9f1a71e1755808c36f not found: ID does not exist" Sep 10 19:45:11.758400 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.758366 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr"] Sep 10 19:45:11.764453 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:11.764420 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1fa9/sub-test-model-2a1fa9-kserve-6bfc8b786c-dw2sr"] Sep 10 19:45:12.489088 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:12.489054 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99061b63-f117-4aa3-914f-99de52674420" path="/var/lib/kubelet/pods/99061b63-f117-4aa3-914f-99de52674420/volumes" Sep 10 19:45:21.399643 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.399595 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn"] Sep 10 19:45:21.400268 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400246 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d9faa3-0e8f-4225-987d-4548a8970ebd" containerName="cleanup" Sep 10 19:45:21.400387 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400307 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d9faa3-0e8f-4225-987d-4548a8970ebd" containerName="cleanup" Sep 10 19:45:21.400387 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400365 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1260fb-9ca8-499c-90f4-b7630ed735ec" containerName="cleanup" Sep 10 19:45:21.400387 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400375 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1260fb-9ca8-499c-90f4-b7630ed735ec" containerName="cleanup" Sep 10 19:45:21.400387 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400387 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d36e27-57a1-492e-9cc4-8da9c0f78b8e" containerName="cleanup" Sep 10 19:45:21.400551 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400395 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d36e27-57a1-492e-9cc4-8da9c0f78b8e" containerName="cleanup" Sep 10 19:45:21.400551 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400423 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99061b63-f117-4aa3-914f-99de52674420" containerName="main" Sep 10 19:45:21.400551 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400432 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="99061b63-f117-4aa3-914f-99de52674420" containerName="main" Sep 10 19:45:21.400551 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400540 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3d36e27-57a1-492e-9cc4-8da9c0f78b8e" containerName="cleanup" Sep 10 19:45:21.400698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400556 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="99061b63-f117-4aa3-914f-99de52674420" containerName="main" Sep 10 19:45:21.400698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400568 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1260fb-9ca8-499c-90f4-b7630ed735ec" containerName="cleanup" Sep 10 19:45:21.400698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.400576 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d9faa3-0e8f-4225-987d-4548a8970ebd" containerName="cleanup" Sep 10 19:45:21.403840 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.403817 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.406221 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.406199 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-261db5-openshift-default-dockercfg-48wv2\"" Sep 10 19:45:21.417991 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.417960 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn"] Sep 10 19:45:21.502416 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.502358 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74l4\" (UniqueName: \"kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-kube-api-access-x74l4\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.502416 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.502422 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-envoy\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.502682 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.502473 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-socket\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.502682 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.502496 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-credential-socket\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.502682 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.502526 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-podinfo\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.502682 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.502549 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-data\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.502682 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.502649 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4882ae-57cd-40d9-93b4-54662af0bb44-istiod-ca-cert\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.502945 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.502691 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-certs\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.502945 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.502722 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-token\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.603681 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.603636 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4882ae-57cd-40d9-93b4-54662af0bb44-istiod-ca-cert\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.603884 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.603695 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-certs\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.603884 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.603729 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-token\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.603884 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.603777 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74l4\" (UniqueName: \"kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-kube-api-access-x74l4\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.604054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.603910 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-envoy\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.604054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.603991 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-socket\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.604054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.604017 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-credential-socket\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.604316 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.604274 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-podinfo\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.604456 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.604432 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-socket\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.604456 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.604450 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-data\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.604643 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.604535 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-credential-socket\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.604643 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.604538 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4882ae-57cd-40d9-93b4-54662af0bb44-istiod-ca-cert\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.604643 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.604301 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-certs\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.604786 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.604673 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-data\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.606570 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.606550 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-envoy\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.606659 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.606597 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-podinfo\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.615837 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.615812 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-token\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.616034 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.616010 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74l4\" (UniqueName: \"kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-kube-api-access-x74l4\") pod \"e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.715812 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.715723 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:21.995340 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.995308 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn"] Sep 10 19:45:21.997213 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:45:21.997186 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4882ae_57cd_40d9_93b4_54662af0bb44.slice/crio-f888e442d818723e739915247caa46eacc874edb021da67e7e39dd938d9d1319 WatchSource:0}: Error finding container f888e442d818723e739915247caa46eacc874edb021da67e7e39dd938d9d1319: Status 404 returned error can't find the container with id f888e442d818723e739915247caa46eacc874edb021da67e7e39dd938d9d1319 Sep 10 19:45:21.999582 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.999546 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:45:21.999673 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.999616 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:45:21.999673 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:21.999644 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:45:22.801172 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:22.801134 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" event={"ID":"5a4882ae-57cd-40d9-93b4-54662af0bb44","Type":"ContainerStarted","Data":"6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa"} Sep 10 19:45:22.801562 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:22.801191 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" event={"ID":"5a4882ae-57cd-40d9-93b4-54662af0bb44","Type":"ContainerStarted","Data":"f888e442d818723e739915247caa46eacc874edb021da67e7e39dd938d9d1319"} Sep 10 19:45:22.830649 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:22.830591 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" podStartSLOduration=1.8305758509999999 podStartE2EDuration="1.830575851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:45:22.830020477 +0000 UTC m=+1508.797832747" watchObservedRunningTime="2026-09-10 19:45:22.830575851 +0000 UTC m=+1508.798388098" Sep 10 19:45:23.716391 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:23.716349 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:23.721912 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:23.721883 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:23.806885 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:23.806848 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:23.807744 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:23.807728 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:26.797199 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:26.797161 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr"] Sep 10 19:45:26.801255 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:26.801228 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:45:26.803778 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:26.803745 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-261db5\"" Sep 10 19:45:26.803945 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:26.803908 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-261db5-dockercfg-dkjpl\"" Sep 10 19:45:26.821479 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:26.817232 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr"] Sep 10 19:45:26.957631 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:26.957585 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c850578-9e62-4836-933e-7e0ed994df12-plugins-config-volume\") pod \"payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr\" (UID: \"9c850578-9e62-4836-933e-7e0ed994df12\") " pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:45:26.957895 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:26.957670 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4psg\" (UniqueName: \"kubernetes.io/projected/9c850578-9e62-4836-933e-7e0ed994df12-kube-api-access-h4psg\") pod \"payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr\" (UID: \"9c850578-9e62-4836-933e-7e0ed994df12\") " pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:45:27.058451 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:27.058411 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4psg\" (UniqueName: \"kubernetes.io/projected/9c850578-9e62-4836-933e-7e0ed994df12-kube-api-access-h4psg\") pod \"payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr\" (UID: \"9c850578-9e62-4836-933e-7e0ed994df12\") " pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:45:27.058654 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:27.058526 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c850578-9e62-4836-933e-7e0ed994df12-plugins-config-volume\") pod \"payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr\" (UID: \"9c850578-9e62-4836-933e-7e0ed994df12\") " pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:45:27.059239 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:27.059218 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c850578-9e62-4836-933e-7e0ed994df12-plugins-config-volume\") pod \"payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr\" (UID: \"9c850578-9e62-4836-933e-7e0ed994df12\") " pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:45:27.068965 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:27.068927 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4psg\" (UniqueName: \"kubernetes.io/projected/9c850578-9e62-4836-933e-7e0ed994df12-kube-api-access-h4psg\") pod \"payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr\" (UID: \"9c850578-9e62-4836-933e-7e0ed994df12\") " pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:45:27.115625 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:27.115579 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:45:27.391754 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:27.391723 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr"] Sep 10 19:45:27.393460 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:45:27.393429 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c850578_9e62_4836_933e_7e0ed994df12.slice/crio-5f48325aaaeafd316246c71e2f6c06345f027196433f596b1c906449906eedb6 WatchSource:0}: Error finding container 5f48325aaaeafd316246c71e2f6c06345f027196433f596b1c906449906eedb6: Status 404 returned error can't find the container with id 5f48325aaaeafd316246c71e2f6c06345f027196433f596b1c906449906eedb6 Sep 10 19:45:27.834158 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:27.834118 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" event={"ID":"9c850578-9e62-4836-933e-7e0ed994df12","Type":"ContainerStarted","Data":"d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9"} Sep 10 19:45:27.834689 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:27.834188 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" event={"ID":"9c850578-9e62-4836-933e-7e0ed994df12","Type":"ContainerStarted","Data":"5f48325aaaeafd316246c71e2f6c06345f027196433f596b1c906449906eedb6"} Sep 10 19:45:27.834689 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:27.834218 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:45:27.853768 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:27.853714 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" podStartSLOduration=1.853695949 podStartE2EDuration="1.853695949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:45:27.851936058 +0000 UTC m=+1513.819748310" watchObservedRunningTime="2026-09-10 19:45:27.853695949 +0000 UTC m=+1513.821508196" Sep 10 19:45:38.842534 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:38.842451 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:45:46.951334 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:46.951296 2584 generic.go:356] "Generic (PLEG): container finished" podID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerID="6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa" exitCode=0 Sep 10 19:45:46.951334 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:46.951313 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" event={"ID":"5a4882ae-57cd-40d9-93b4-54662af0bb44","Type":"ContainerDied","Data":"6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa"} Sep 10 19:45:46.951876 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:46.951725 2584 scope.go:117] "RemoveContainer" containerID="6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa" Sep 10 19:45:46.952910 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:46.952880 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:45:46.952973 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:46.952945 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:45:46.953012 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:46.952976 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 19:45:47.959982 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:47.959948 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" event={"ID":"5a4882ae-57cd-40d9-93b4-54662af0bb44","Type":"ContainerStarted","Data":"05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019"} Sep 10 19:45:48.717328 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:48.717263 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:48.717562 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:48.717426 2584 patch_prober.go:28] interesting pod/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 19:45:48.717562 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:48.717475 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 19:45:49.717210 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:49.717165 2584 patch_prober.go:28] interesting pod/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 19:45:49.717871 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:49.717834 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 19:45:50.717359 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:50.717316 2584 patch_prober.go:28] interesting pod/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 19:45:50.718018 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:50.717981 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 19:45:51.716647 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:51.716599 2584 patch_prober.go:28] interesting pod/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 19:45:51.716880 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:51.716680 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 19:45:52.716864 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:52.716820 2584 patch_prober.go:28] interesting pod/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 19:45:52.717359 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:52.716887 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 19:45:53.721192 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:53.721156 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:53.721593 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:53.721510 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:45:53.722350 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:45:53.722332 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:46:14.657913 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:14.657877 2584 scope.go:117] "RemoveContainer" containerID="718042b6b2c05ff4d0d2fdc83fcbeac376b5c9530c4954f8683d55155bc4a8a2" Sep 10 19:46:31.925165 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:31.925120 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr"] Sep 10 19:46:31.925580 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:31.925418 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" podUID="9c850578-9e62-4836-933e-7e0ed994df12" containerName="payload-processing" containerID="cri-o://d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9" gracePeriod=30 Sep 10 19:46:32.184341 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.184240 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:46:32.195140 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.195104 2584 generic.go:356] "Generic (PLEG): container finished" podID="9c850578-9e62-4836-933e-7e0ed994df12" containerID="d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9" exitCode=0 Sep 10 19:46:32.195348 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.195155 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" Sep 10 19:46:32.195348 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.195197 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" event={"ID":"9c850578-9e62-4836-933e-7e0ed994df12","Type":"ContainerDied","Data":"d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9"} Sep 10 19:46:32.195348 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.195303 2584 scope.go:117] "RemoveContainer" containerID="d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9" Sep 10 19:46:32.195505 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.195304 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr" event={"ID":"9c850578-9e62-4836-933e-7e0ed994df12","Type":"ContainerDied","Data":"5f48325aaaeafd316246c71e2f6c06345f027196433f596b1c906449906eedb6"} Sep 10 19:46:32.204402 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.204335 2584 scope.go:117] "RemoveContainer" containerID="d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9" Sep 10 19:46:32.204661 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:46:32.204640 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9\": container with ID starting with d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9 not found: ID does not exist" containerID="d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9" Sep 10 19:46:32.204723 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.204669 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9"} err="failed to get container status \"d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9\": rpc error: code = NotFound desc = could not find container \"d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9\": container with ID starting with d2a141130176463ccc4797db6a9055327f9d6596796d38c8176f010edb17a2e9 not found: ID does not exist" Sep 10 19:46:32.307671 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.307636 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4psg\" (UniqueName: \"kubernetes.io/projected/9c850578-9e62-4836-933e-7e0ed994df12-kube-api-access-h4psg\") pod \"9c850578-9e62-4836-933e-7e0ed994df12\" (UID: \"9c850578-9e62-4836-933e-7e0ed994df12\") " Sep 10 19:46:32.307886 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.307803 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c850578-9e62-4836-933e-7e0ed994df12-plugins-config-volume\") pod \"9c850578-9e62-4836-933e-7e0ed994df12\" (UID: \"9c850578-9e62-4836-933e-7e0ed994df12\") " Sep 10 19:46:32.308186 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.308156 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c850578-9e62-4836-933e-7e0ed994df12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c850578-9e62-4836-933e-7e0ed994df12" (UID: "9c850578-9e62-4836-933e-7e0ed994df12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:46:32.309749 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.309719 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c850578-9e62-4836-933e-7e0ed994df12-kube-api-access-h4psg" (OuterVolumeSpecName: "kube-api-access-h4psg") pod "9c850578-9e62-4836-933e-7e0ed994df12" (UID: "9c850578-9e62-4836-933e-7e0ed994df12"). InnerVolumeSpecName "kube-api-access-h4psg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:46:32.409524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.409474 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4psg\" (UniqueName: \"kubernetes.io/projected/9c850578-9e62-4836-933e-7e0ed994df12-kube-api-access-h4psg\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:32.409524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.409515 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c850578-9e62-4836-933e-7e0ed994df12-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:32.519903 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.519866 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr"] Sep 10 19:46:32.523200 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:32.523168 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-261db5-7495fb8cb9-pkhfr"] Sep 10 19:46:34.488906 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:34.488871 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c850578-9e62-4836-933e-7e0ed994df12" path="/var/lib/kubelet/pods/9c850578-9e62-4836-933e-7e0ed994df12/volumes" Sep 10 19:46:38.797057 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:38.797019 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn"] Sep 10 19:46:38.797474 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:38.797224 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" containerID="cri-o://05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019" gracePeriod=30 Sep 10 19:46:44.051648 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.051621 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:46:44.234531 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234422 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-data\") pod \"5a4882ae-57cd-40d9-93b4-54662af0bb44\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " Sep 10 19:46:44.234531 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234484 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4882ae-57cd-40d9-93b4-54662af0bb44-istiod-ca-cert\") pod \"5a4882ae-57cd-40d9-93b4-54662af0bb44\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " Sep 10 19:46:44.234531 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234507 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-token\") pod \"5a4882ae-57cd-40d9-93b4-54662af0bb44\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " Sep 10 19:46:44.234531 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234527 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-socket\") pod \"5a4882ae-57cd-40d9-93b4-54662af0bb44\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " Sep 10 19:46:44.234872 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234566 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-certs\") pod \"5a4882ae-57cd-40d9-93b4-54662af0bb44\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " Sep 10 19:46:44.234872 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234588 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-envoy\") pod \"5a4882ae-57cd-40d9-93b4-54662af0bb44\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " Sep 10 19:46:44.234872 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234796 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a4882ae-57cd-40d9-93b4-54662af0bb44" (UID: "5a4882ae-57cd-40d9-93b4-54662af0bb44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:46:44.235026 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234919 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a4882ae-57cd-40d9-93b4-54662af0bb44" (UID: "5a4882ae-57cd-40d9-93b4-54662af0bb44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:46:44.235026 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234936 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-podinfo\") pod \"5a4882ae-57cd-40d9-93b4-54662af0bb44\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " Sep 10 19:46:44.235026 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234937 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a4882ae-57cd-40d9-93b4-54662af0bb44" (UID: "5a4882ae-57cd-40d9-93b4-54662af0bb44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:46:44.235026 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.234980 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-credential-socket\") pod \"5a4882ae-57cd-40d9-93b4-54662af0bb44\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " Sep 10 19:46:44.235266 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.235018 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4882ae-57cd-40d9-93b4-54662af0bb44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a4882ae-57cd-40d9-93b4-54662af0bb44" (UID: "5a4882ae-57cd-40d9-93b4-54662af0bb44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:46:44.235266 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.235031 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x74l4\" (UniqueName: \"kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-kube-api-access-x74l4\") pod \"5a4882ae-57cd-40d9-93b4-54662af0bb44\" (UID: \"5a4882ae-57cd-40d9-93b4-54662af0bb44\") " Sep 10 19:46:44.235404 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.235274 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a4882ae-57cd-40d9-93b4-54662af0bb44" (UID: "5a4882ae-57cd-40d9-93b4-54662af0bb44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:46:44.235497 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.235458 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-data\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:44.235497 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.235484 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4882ae-57cd-40d9-93b4-54662af0bb44-istiod-ca-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:44.235497 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.235498 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:44.235736 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.235511 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-workload-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:44.235736 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.235524 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-credential-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:44.236969 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.236942 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a4882ae-57cd-40d9-93b4-54662af0bb44" (UID: "5a4882ae-57cd-40d9-93b4-54662af0bb44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:46:44.237084 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.237065 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a4882ae-57cd-40d9-93b4-54662af0bb44" (UID: "5a4882ae-57cd-40d9-93b4-54662af0bb44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:46:44.237357 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.237338 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a4882ae-57cd-40d9-93b4-54662af0bb44" (UID: "5a4882ae-57cd-40d9-93b4-54662af0bb44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:46:44.237410 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.237384 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-kube-api-access-x74l4" (OuterVolumeSpecName: "kube-api-access-x74l4") pod "5a4882ae-57cd-40d9-93b4-54662af0bb44" (UID: "5a4882ae-57cd-40d9-93b4-54662af0bb44"). InnerVolumeSpecName "kube-api-access-x74l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:46:44.264235 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.264202 2584 generic.go:356] "Generic (PLEG): container finished" podID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerID="05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019" exitCode=0 Sep 10 19:46:44.264422 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.264253 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" Sep 10 19:46:44.264422 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.264293 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" event={"ID":"5a4882ae-57cd-40d9-93b4-54662af0bb44","Type":"ContainerDied","Data":"05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019"} Sep 10 19:46:44.264422 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.264391 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn" event={"ID":"5a4882ae-57cd-40d9-93b4-54662af0bb44","Type":"ContainerDied","Data":"f888e442d818723e739915247caa46eacc874edb021da67e7e39dd938d9d1319"} Sep 10 19:46:44.264422 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.264419 2584 scope.go:117] "RemoveContainer" containerID="05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019" Sep 10 19:46:44.274452 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.274429 2584 scope.go:117] "RemoveContainer" containerID="6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa" Sep 10 19:46:44.283379 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.283359 2584 scope.go:117] "RemoveContainer" containerID="05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019" Sep 10 19:46:44.283728 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:46:44.283705 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019\": container with ID starting with 05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019 not found: ID does not exist" containerID="05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019" Sep 10 19:46:44.283774 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.283739 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019"} err="failed to get container status \"05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019\": rpc error: code = NotFound desc = could not find container \"05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019\": container with ID starting with 05584cd8f9037c300dce45ca348dbc829fda33c20e8f19c95f7b554ab66a5019 not found: ID does not exist" Sep 10 19:46:44.283774 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.283757 2584 scope.go:117] "RemoveContainer" containerID="6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa" Sep 10 19:46:44.284033 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:46:44.284014 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa\": container with ID starting with 6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa not found: ID does not exist" containerID="6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa" Sep 10 19:46:44.284097 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.284039 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa"} err="failed to get container status \"6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa\": rpc error: code = NotFound desc = could not find container \"6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa\": container with ID starting with 6806a8c828b2bf73999984d2a88681b0e671986b3ecad9f45aeb7c73c898c7aa not found: ID does not exist" Sep 10 19:46:44.288716 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.288690 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn"] Sep 10 19:46:44.294525 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.294497 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-261db5-openshift-default-644cc566c8-6n4mn"] Sep 10 19:46:44.336583 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.336543 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-token\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:44.336583 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.336577 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-envoy\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:44.336583 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.336586 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4882ae-57cd-40d9-93b4-54662af0bb44-istio-podinfo\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:44.336830 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.336595 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x74l4\" (UniqueName: \"kubernetes.io/projected/5a4882ae-57cd-40d9-93b4-54662af0bb44-kube-api-access-x74l4\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:44.489960 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:44.489874 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" path="/var/lib/kubelet/pods/5a4882ae-57cd-40d9-93b4-54662af0bb44/volumes" Sep 10 19:46:49.277923 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:49.277878 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw"] Sep 10 19:46:49.278536 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:49.278181 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" podUID="db1df488-399f-4626-b620-8d2ae77848e2" containerName="payload-pre-processing" containerID="cri-o://3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3" gracePeriod=30 Sep 10 19:46:49.541499 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:49.541431 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:46:49.592626 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:49.592585 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1df488-399f-4626-b620-8d2ae77848e2-plugins-config-volume\") pod \"db1df488-399f-4626-b620-8d2ae77848e2\" (UID: \"db1df488-399f-4626-b620-8d2ae77848e2\") " Sep 10 19:46:49.592836 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:49.592650 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx4xw\" (UniqueName: \"kubernetes.io/projected/db1df488-399f-4626-b620-8d2ae77848e2-kube-api-access-lx4xw\") pod \"db1df488-399f-4626-b620-8d2ae77848e2\" (UID: \"db1df488-399f-4626-b620-8d2ae77848e2\") " Sep 10 19:46:49.593007 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:49.592979 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1df488-399f-4626-b620-8d2ae77848e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db1df488-399f-4626-b620-8d2ae77848e2" (UID: "db1df488-399f-4626-b620-8d2ae77848e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:46:49.594804 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:49.594757 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1df488-399f-4626-b620-8d2ae77848e2-kube-api-access-lx4xw" (OuterVolumeSpecName: "kube-api-access-lx4xw") pod "db1df488-399f-4626-b620-8d2ae77848e2" (UID: "db1df488-399f-4626-b620-8d2ae77848e2"). InnerVolumeSpecName "kube-api-access-lx4xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:46:49.694153 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:49.694107 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1df488-399f-4626-b620-8d2ae77848e2-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:49.694153 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:49.694141 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx4xw\" (UniqueName: \"kubernetes.io/projected/db1df488-399f-4626-b620-8d2ae77848e2-kube-api-access-lx4xw\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:46:50.300818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:50.300777 2584 generic.go:356] "Generic (PLEG): container finished" podID="db1df488-399f-4626-b620-8d2ae77848e2" containerID="3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3" exitCode=0 Sep 10 19:46:50.301323 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:50.300826 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" Sep 10 19:46:50.301323 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:50.300874 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" event={"ID":"db1df488-399f-4626-b620-8d2ae77848e2","Type":"ContainerDied","Data":"3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3"} Sep 10 19:46:50.301323 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:50.300945 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw" event={"ID":"db1df488-399f-4626-b620-8d2ae77848e2","Type":"ContainerDied","Data":"3a422b9b2d79ff2b07e42b7b20cafc32c9498cdb8ef3793b57ec666b3fbfde63"} Sep 10 19:46:50.301323 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:50.300981 2584 scope.go:117] "RemoveContainer" containerID="3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3" Sep 10 19:46:50.310292 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:50.310256 2584 scope.go:117] "RemoveContainer" containerID="3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3" Sep 10 19:46:50.310628 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:46:50.310609 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3\": container with ID starting with 3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3 not found: ID does not exist" containerID="3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3" Sep 10 19:46:50.310676 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:50.310637 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3"} err="failed to get container status \"3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3\": rpc error: code = NotFound desc = could not find container \"3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3\": container with ID starting with 3d5ceb095d0fdd1a88526befd872dfca4d05444910cf5a49c2787f16c952f1e3 not found: ID does not exist" Sep 10 19:46:50.324577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:50.324529 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw"] Sep 10 19:46:50.332660 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:50.332628 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ecadb8-6f7474798b-nnzbw"] Sep 10 19:46:50.488118 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:50.488082 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db1df488-399f-4626-b620-8d2ae77848e2" path="/var/lib/kubelet/pods/db1df488-399f-4626-b620-8d2ae77848e2/volumes" Sep 10 19:46:56.349031 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.348992 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj"] Sep 10 19:46:56.349449 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349425 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1df488-399f-4626-b620-8d2ae77848e2" containerName="payload-pre-processing" Sep 10 19:46:56.349449 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349439 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1df488-399f-4626-b620-8d2ae77848e2" containerName="payload-pre-processing" Sep 10 19:46:56.349524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349451 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c850578-9e62-4836-933e-7e0ed994df12" containerName="payload-processing" Sep 10 19:46:56.349524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349457 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c850578-9e62-4836-933e-7e0ed994df12" containerName="payload-processing" Sep 10 19:46:56.349524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349466 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" Sep 10 19:46:56.349524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349473 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" Sep 10 19:46:56.349524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349481 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" Sep 10 19:46:56.349524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349486 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" Sep 10 19:46:56.349710 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349550 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1df488-399f-4626-b620-8d2ae77848e2" containerName="payload-pre-processing" Sep 10 19:46:56.349710 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349560 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" Sep 10 19:46:56.349710 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349566 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c850578-9e62-4836-933e-7e0ed994df12" containerName="payload-processing" Sep 10 19:46:56.349710 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.349572 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4882ae-57cd-40d9-93b4-54662af0bb44" containerName="istio-proxy" Sep 10 19:46:56.352797 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.352777 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:46:56.355138 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.355111 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b588eb\"" Sep 10 19:46:56.367004 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.366970 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj"] Sep 10 19:46:56.373905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.373876 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq"] Sep 10 19:46:56.376428 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.376406 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:46:56.378937 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.378909 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b588eb-dockercfg-bndgq\"" Sep 10 19:46:56.389181 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.389152 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq"] Sep 10 19:46:56.459769 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.459711 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c51f02-21c3-481f-b360-374a0f9d6973-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq\" (UID: \"11c51f02-21c3-481f-b360-374a0f9d6973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:46:56.459963 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.459775 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72c938c-ce6d-498e-b164-8ccff23c7df9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj\" (UID: \"b72c938c-ce6d-498e-b164-8ccff23c7df9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:46:56.459963 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.459827 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrwd9\" (UniqueName: \"kubernetes.io/projected/b72c938c-ce6d-498e-b164-8ccff23c7df9-kube-api-access-zrwd9\") pod \"payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj\" (UID: \"b72c938c-ce6d-498e-b164-8ccff23c7df9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:46:56.459963 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.459898 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bpm8\" (UniqueName: \"kubernetes.io/projected/11c51f02-21c3-481f-b360-374a0f9d6973-kube-api-access-4bpm8\") pod \"payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq\" (UID: \"11c51f02-21c3-481f-b360-374a0f9d6973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:46:56.560844 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.560801 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c51f02-21c3-481f-b360-374a0f9d6973-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq\" (UID: \"11c51f02-21c3-481f-b360-374a0f9d6973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:46:56.561022 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.560858 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72c938c-ce6d-498e-b164-8ccff23c7df9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj\" (UID: \"b72c938c-ce6d-498e-b164-8ccff23c7df9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:46:56.561022 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.560970 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrwd9\" (UniqueName: \"kubernetes.io/projected/b72c938c-ce6d-498e-b164-8ccff23c7df9-kube-api-access-zrwd9\") pod \"payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj\" (UID: \"b72c938c-ce6d-498e-b164-8ccff23c7df9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:46:56.561123 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.561048 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bpm8\" (UniqueName: \"kubernetes.io/projected/11c51f02-21c3-481f-b360-374a0f9d6973-kube-api-access-4bpm8\") pod \"payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq\" (UID: \"11c51f02-21c3-481f-b360-374a0f9d6973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:46:56.561816 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.561791 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72c938c-ce6d-498e-b164-8ccff23c7df9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj\" (UID: \"b72c938c-ce6d-498e-b164-8ccff23c7df9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:46:56.561904 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.561791 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c51f02-21c3-481f-b360-374a0f9d6973-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq\" (UID: \"11c51f02-21c3-481f-b360-374a0f9d6973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:46:56.570075 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.570050 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrwd9\" (UniqueName: \"kubernetes.io/projected/b72c938c-ce6d-498e-b164-8ccff23c7df9-kube-api-access-zrwd9\") pod \"payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj\" (UID: \"b72c938c-ce6d-498e-b164-8ccff23c7df9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:46:56.570194 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.570176 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bpm8\" (UniqueName: \"kubernetes.io/projected/11c51f02-21c3-481f-b360-374a0f9d6973-kube-api-access-4bpm8\") pod \"payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq\" (UID: \"11c51f02-21c3-481f-b360-374a0f9d6973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:46:56.663254 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.663148 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:46:56.686935 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.686895 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:46:56.949870 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.949841 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj"] Sep 10 19:46:56.952954 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:46:56.952917 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72c938c_ce6d_498e_b164_8ccff23c7df9.slice/crio-45ec6e7b901842365a2a33564f8c501616c4c4d0672e17c31c61798bd0dc7ace WatchSource:0}: Error finding container 45ec6e7b901842365a2a33564f8c501616c4c4d0672e17c31c61798bd0dc7ace: Status 404 returned error can't find the container with id 45ec6e7b901842365a2a33564f8c501616c4c4d0672e17c31c61798bd0dc7ace Sep 10 19:46:56.976953 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:56.976921 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq"] Sep 10 19:46:56.988499 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:46:56.988465 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c51f02_21c3_481f_b360_374a0f9d6973.slice/crio-c61d33dbd734701b8578771f390b5a1ad9d5dce05bf87757757b838d60dc1b9a WatchSource:0}: Error finding container c61d33dbd734701b8578771f390b5a1ad9d5dce05bf87757757b838d60dc1b9a: Status 404 returned error can't find the container with id c61d33dbd734701b8578771f390b5a1ad9d5dce05bf87757757b838d60dc1b9a Sep 10 19:46:57.342743 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:57.342702 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" event={"ID":"11c51f02-21c3-481f-b360-374a0f9d6973","Type":"ContainerStarted","Data":"8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d"} Sep 10 19:46:57.342944 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:57.342778 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:46:57.342944 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:57.342806 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" event={"ID":"11c51f02-21c3-481f-b360-374a0f9d6973","Type":"ContainerStarted","Data":"c61d33dbd734701b8578771f390b5a1ad9d5dce05bf87757757b838d60dc1b9a"} Sep 10 19:46:57.344558 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:57.344528 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" event={"ID":"b72c938c-ce6d-498e-b164-8ccff23c7df9","Type":"ContainerStarted","Data":"a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b"} Sep 10 19:46:57.344689 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:57.344582 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" event={"ID":"b72c938c-ce6d-498e-b164-8ccff23c7df9","Type":"ContainerStarted","Data":"45ec6e7b901842365a2a33564f8c501616c4c4d0672e17c31c61798bd0dc7ace"} Sep 10 19:46:57.344689 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:57.344611 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:46:57.367145 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:57.367083 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" podStartSLOduration=1.367067094 podStartE2EDuration="1.367067094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:46:57.365182344 +0000 UTC m=+1603.332994592" watchObservedRunningTime="2026-09-10 19:46:57.367067094 +0000 UTC m=+1603.334879340" Sep 10 19:46:57.387170 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:46:57.387112 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" podStartSLOduration=1.387093187 podStartE2EDuration="1.387093187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:46:57.385419109 +0000 UTC m=+1603.353231356" watchObservedRunningTime="2026-09-10 19:46:57.387093187 +0000 UTC m=+1603.354905433" Sep 10 19:47:08.291727 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.291639 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89"] Sep 10 19:47:08.295544 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.295511 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff3019-29817825-xvx89"] Sep 10 19:47:08.328937 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.328898 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2"] Sep 10 19:47:08.329209 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.329177 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" podUID="2632965c-2476-4d1a-96ef-4f8dc3cfb8ae" containerName="payload-processing" containerID="cri-o://2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6" gracePeriod=30 Sep 10 19:47:08.337961 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.337865 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s"] Sep 10 19:47:08.338144 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.338104 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" podUID="44a2f13c-4814-4ab1-9046-fa84bb1ca5ca" containerName="payload-pre-processing" containerID="cri-o://029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e" gracePeriod=30 Sep 10 19:47:08.355073 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.354332 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:47:08.355240 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.355106 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:47:08.489637 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.489596 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee1260fb-9ca8-499c-90f4-b7630ed735ec" path="/var/lib/kubelet/pods/ee1260fb-9ca8-499c-90f4-b7630ed735ec/volumes" Sep 10 19:47:08.615707 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.615675 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:47:08.618817 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.618792 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:47:08.689119 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.684444 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-plugins-config-volume\") pod \"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca\" (UID: \"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca\") " Sep 10 19:47:08.689119 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.684562 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swb9l\" (UniqueName: \"kubernetes.io/projected/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-kube-api-access-swb9l\") pod \"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae\" (UID: \"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae\") " Sep 10 19:47:08.689119 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.684599 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-plugins-config-volume\") pod \"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae\" (UID: \"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae\") " Sep 10 19:47:08.689119 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.684641 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5fcp\" (UniqueName: \"kubernetes.io/projected/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-kube-api-access-c5fcp\") pod \"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca\" (UID: \"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca\") " Sep 10 19:47:08.689119 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.686135 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2632965c-2476-4d1a-96ef-4f8dc3cfb8ae" (UID: "2632965c-2476-4d1a-96ef-4f8dc3cfb8ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:47:08.689119 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.686576 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44a2f13c-4814-4ab1-9046-fa84bb1ca5ca" (UID: "44a2f13c-4814-4ab1-9046-fa84bb1ca5ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:47:08.698112 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.690723 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-kube-api-access-c5fcp" (OuterVolumeSpecName: "kube-api-access-c5fcp") pod "44a2f13c-4814-4ab1-9046-fa84bb1ca5ca" (UID: "44a2f13c-4814-4ab1-9046-fa84bb1ca5ca"). InnerVolumeSpecName "kube-api-access-c5fcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:47:08.701562 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.701523 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-kube-api-access-swb9l" (OuterVolumeSpecName: "kube-api-access-swb9l") pod "2632965c-2476-4d1a-96ef-4f8dc3cfb8ae" (UID: "2632965c-2476-4d1a-96ef-4f8dc3cfb8ae"). InnerVolumeSpecName "kube-api-access-swb9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:47:08.785794 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.785748 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swb9l\" (UniqueName: \"kubernetes.io/projected/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-kube-api-access-swb9l\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:08.785794 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.785785 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:08.785794 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.785795 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5fcp\" (UniqueName: \"kubernetes.io/projected/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-kube-api-access-c5fcp\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:08.785794 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:08.785806 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:09.414493 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.414453 2584 generic.go:356] "Generic (PLEG): container finished" podID="2632965c-2476-4d1a-96ef-4f8dc3cfb8ae" containerID="2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6" exitCode=0 Sep 10 19:47:09.414973 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.414528 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" Sep 10 19:47:09.414973 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.414537 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" event={"ID":"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae","Type":"ContainerDied","Data":"2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6"} Sep 10 19:47:09.414973 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.414634 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2" event={"ID":"2632965c-2476-4d1a-96ef-4f8dc3cfb8ae","Type":"ContainerDied","Data":"72c71fcfe2f9a23755deeca3003a74465b32fb4e5e51c88563801934a185c21e"} Sep 10 19:47:09.414973 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.414670 2584 scope.go:117] "RemoveContainer" containerID="2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6" Sep 10 19:47:09.416025 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.415970 2584 generic.go:356] "Generic (PLEG): container finished" podID="44a2f13c-4814-4ab1-9046-fa84bb1ca5ca" containerID="029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e" exitCode=0 Sep 10 19:47:09.416025 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.416004 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" event={"ID":"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca","Type":"ContainerDied","Data":"029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e"} Sep 10 19:47:09.416152 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.416039 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" Sep 10 19:47:09.416152 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.416062 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s" event={"ID":"44a2f13c-4814-4ab1-9046-fa84bb1ca5ca","Type":"ContainerDied","Data":"8b08006cba4082d3464af2af1a03aee10d7f67b40fc8ed78be7edadfe7fb4ade"} Sep 10 19:47:09.425203 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.425179 2584 scope.go:117] "RemoveContainer" containerID="2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6" Sep 10 19:47:09.425552 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:47:09.425525 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6\": container with ID starting with 2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6 not found: ID does not exist" containerID="2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6" Sep 10 19:47:09.425657 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.425565 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6"} err="failed to get container status \"2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6\": rpc error: code = NotFound desc = could not find container \"2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6\": container with ID starting with 2f5ea2c836235d2cc5bb5bd4b65e5df06ee6d6297bda4601707b92efec06aea6 not found: ID does not exist" Sep 10 19:47:09.425657 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.425591 2584 scope.go:117] "RemoveContainer" containerID="029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e" Sep 10 19:47:09.433579 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.433560 2584 scope.go:117] "RemoveContainer" containerID="029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e" Sep 10 19:47:09.433874 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:47:09.433851 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e\": container with ID starting with 029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e not found: ID does not exist" containerID="029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e" Sep 10 19:47:09.433927 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.433884 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e"} err="failed to get container status \"029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e\": rpc error: code = NotFound desc = could not find container \"029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e\": container with ID starting with 029542ec642903da5dfe1b57025519b5a6793695876dc40381a15434ecd1606e not found: ID does not exist" Sep 10 19:47:09.438419 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.438395 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2"] Sep 10 19:47:09.445153 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.445129 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff3019-6f48ddc969-csms2"] Sep 10 19:47:09.458250 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.458217 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s"] Sep 10 19:47:09.464767 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:09.464732 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff3019-b4695db6b-5fv5s"] Sep 10 19:47:10.488674 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:10.488628 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2632965c-2476-4d1a-96ef-4f8dc3cfb8ae" path="/var/lib/kubelet/pods/2632965c-2476-4d1a-96ef-4f8dc3cfb8ae/volumes" Sep 10 19:47:10.489097 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:10.489042 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a2f13c-4814-4ab1-9046-fa84bb1ca5ca" path="/var/lib/kubelet/pods/44a2f13c-4814-4ab1-9046-fa84bb1ca5ca/volumes" Sep 10 19:47:15.203815 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:15.203778 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb"] Sep 10 19:47:15.204328 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:15.204035 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" containerID="cri-o://61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2" gracePeriod=30 Sep 10 19:47:20.461131 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.461105 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:47:20.476596 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.476509 2584 generic.go:356] "Generic (PLEG): container finished" podID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerID="61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2" exitCode=0 Sep 10 19:47:20.476596 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.476561 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" Sep 10 19:47:20.476827 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.476589 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" event={"ID":"ba1bb938-a6d4-42c7-8771-a82c6bcf536d","Type":"ContainerDied","Data":"61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2"} Sep 10 19:47:20.476827 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.476650 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb" event={"ID":"ba1bb938-a6d4-42c7-8771-a82c6bcf536d","Type":"ContainerDied","Data":"6747296fc9e68bf0d5158b633dc67238bb26e6fff66dcc7ac94e5e50b50a21ad"} Sep 10 19:47:20.476827 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.476679 2584 scope.go:117] "RemoveContainer" containerID="61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2" Sep 10 19:47:20.487169 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.485998 2584 scope.go:117] "RemoveContainer" containerID="5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d" Sep 10 19:47:20.502815 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.502789 2584 scope.go:117] "RemoveContainer" containerID="61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2" Sep 10 19:47:20.503128 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:47:20.503102 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2\": container with ID starting with 61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2 not found: ID does not exist" containerID="61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2" Sep 10 19:47:20.503211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.503136 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2"} err="failed to get container status \"61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2\": rpc error: code = NotFound desc = could not find container \"61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2\": container with ID starting with 61652a97ae33c1dd4eca708c53e5b530a207f51af2f3e7aea8afefe0f89b57c2 not found: ID does not exist" Sep 10 19:47:20.503211 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.503155 2584 scope.go:117] "RemoveContainer" containerID="5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d" Sep 10 19:47:20.503496 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:47:20.503474 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d\": container with ID starting with 5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d not found: ID does not exist" containerID="5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d" Sep 10 19:47:20.503589 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.503499 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d"} err="failed to get container status \"5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d\": rpc error: code = NotFound desc = could not find container \"5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d\": container with ID starting with 5ad2f7c81349ebf3cf41dd1ba3838edf8cfa0ebd1fcf7dcb3cdbee062c4edb2d not found: ID does not exist" Sep 10 19:47:20.609819 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.609779 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-certs\") pod \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " Sep 10 19:47:20.609819 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.609825 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-socket\") pod \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " Sep 10 19:47:20.610137 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.609854 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zttdh\" (UniqueName: \"kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-kube-api-access-zttdh\") pod \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " Sep 10 19:47:20.610137 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.609911 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-podinfo\") pod \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " Sep 10 19:47:20.610137 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.609943 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-credential-socket\") pod \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " Sep 10 19:47:20.610137 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610005 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-envoy\") pod \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " Sep 10 19:47:20.610137 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610061 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istiod-ca-cert\") pod \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " Sep 10 19:47:20.610137 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610086 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-data\") pod \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " Sep 10 19:47:20.610137 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610123 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-token\") pod \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\" (UID: \"ba1bb938-a6d4-42c7-8771-a82c6bcf536d\") " Sep 10 19:47:20.610533 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610178 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba1bb938-a6d4-42c7-8771-a82c6bcf536d" (UID: "ba1bb938-a6d4-42c7-8771-a82c6bcf536d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:47:20.610533 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610212 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba1bb938-a6d4-42c7-8771-a82c6bcf536d" (UID: "ba1bb938-a6d4-42c7-8771-a82c6bcf536d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:47:20.610533 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610352 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba1bb938-a6d4-42c7-8771-a82c6bcf536d" (UID: "ba1bb938-a6d4-42c7-8771-a82c6bcf536d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:47:20.610696 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610618 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba1bb938-a6d4-42c7-8771-a82c6bcf536d" (UID: "ba1bb938-a6d4-42c7-8771-a82c6bcf536d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:47:20.610754 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610729 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba1bb938-a6d4-42c7-8771-a82c6bcf536d" (UID: "ba1bb938-a6d4-42c7-8771-a82c6bcf536d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:47:20.610877 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610838 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istiod-ca-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:20.610877 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610866 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-data\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:20.610877 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610879 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:20.611070 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610894 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-workload-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:20.611070 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.610906 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-credential-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:20.612380 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.612353 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-kube-api-access-zttdh" (OuterVolumeSpecName: "kube-api-access-zttdh") pod "ba1bb938-a6d4-42c7-8771-a82c6bcf536d" (UID: "ba1bb938-a6d4-42c7-8771-a82c6bcf536d"). InnerVolumeSpecName "kube-api-access-zttdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:47:20.612500 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.612477 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba1bb938-a6d4-42c7-8771-a82c6bcf536d" (UID: "ba1bb938-a6d4-42c7-8771-a82c6bcf536d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:47:20.612612 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.612598 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba1bb938-a6d4-42c7-8771-a82c6bcf536d" (UID: "ba1bb938-a6d4-42c7-8771-a82c6bcf536d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:47:20.612656 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.612604 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba1bb938-a6d4-42c7-8771-a82c6bcf536d" (UID: "ba1bb938-a6d4-42c7-8771-a82c6bcf536d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:47:20.711818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.711775 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zttdh\" (UniqueName: \"kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-kube-api-access-zttdh\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:20.711818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.711813 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-podinfo\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:20.711818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.711825 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-envoy\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:20.711818 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.711833 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1bb938-a6d4-42c7-8771-a82c6bcf536d-istio-token\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:47:20.800446 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.800409 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb"] Sep 10 19:47:20.806915 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:20.806885 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff3019-openshift-default-cfbb9cbb6-hmzhb"] Sep 10 19:47:22.489088 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:22.489052 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" path="/var/lib/kubelet/pods/ba1bb938-a6d4-42c7-8771-a82c6bcf536d/volumes" Sep 10 19:47:24.492158 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:24.492113 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq_11c51f02-21c3-481f-b360-374a0f9d6973/payload-processing/0.log" Sep 10 19:47:41.209887 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.209849 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8"] Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210237 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a2f13c-4814-4ab1-9046-fa84bb1ca5ca" containerName="payload-pre-processing" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210248 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a2f13c-4814-4ab1-9046-fa84bb1ca5ca" containerName="payload-pre-processing" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210260 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210265 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210314 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210321 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210328 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2632965c-2476-4d1a-96ef-4f8dc3cfb8ae" containerName="payload-processing" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210333 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2632965c-2476-4d1a-96ef-4f8dc3cfb8ae" containerName="payload-processing" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210386 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2632965c-2476-4d1a-96ef-4f8dc3cfb8ae" containerName="payload-processing" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210394 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210402 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba1bb938-a6d4-42c7-8771-a82c6bcf536d" containerName="istio-proxy" Sep 10 19:47:41.210430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.210409 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a2f13c-4814-4ab1-9046-fa84bb1ca5ca" containerName="payload-pre-processing" Sep 10 19:47:41.213515 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.213492 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.217652 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.217621 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-836291\"/\"openshift-service-ca.crt\"" Sep 10 19:47:41.217801 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.217660 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-836291\"/\"ipp-route-836291-kserve-self-signed-certs\"" Sep 10 19:47:41.217801 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.217659 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-836291\"/\"default-dockercfg-v494x\"" Sep 10 19:47:41.217801 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.217787 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-836291\"/\"kube-root-ca.crt\"" Sep 10 19:47:41.229792 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.229761 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8"] Sep 10 19:47:41.293179 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.293134 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-home\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.293409 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.293189 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-model-cache\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.293409 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.293220 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tls-certs\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.293409 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.293235 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9q6c\" (UniqueName: \"kubernetes.io/projected/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-kube-api-access-g9q6c\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.293409 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.293268 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tmp-dir\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.293409 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.293380 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-dshm\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.394117 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.394059 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-model-cache\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.394117 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.394132 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tls-certs\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.394422 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.394162 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9q6c\" (UniqueName: \"kubernetes.io/projected/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-kube-api-access-g9q6c\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.394422 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.394196 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tmp-dir\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.394422 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.394263 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-dshm\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.394422 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.394364 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-home\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.394642 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.394591 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-model-cache\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.394642 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.394614 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tmp-dir\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.394642 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.394633 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-home\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.396635 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.396609 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-dshm\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.396800 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.396781 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tls-certs\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.406236 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.406210 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9q6c\" (UniqueName: \"kubernetes.io/projected/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-kube-api-access-g9q6c\") pod \"ipp-route-836291-kserve-86c8785675-vj2k8\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.524393 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.524293 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:41.805041 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:41.805012 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8"] Sep 10 19:47:41.807149 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:47:41.807123 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf67b152b_5cc3_4ae2_bcfb_6c16f511a340.slice/crio-e077076262ea85e1e8c9ccce3c122741a07fb81938ecd5142b96f2b056fbaf0b WatchSource:0}: Error finding container e077076262ea85e1e8c9ccce3c122741a07fb81938ecd5142b96f2b056fbaf0b: Status 404 returned error can't find the container with id e077076262ea85e1e8c9ccce3c122741a07fb81938ecd5142b96f2b056fbaf0b Sep 10 19:47:42.591473 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:42.591435 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" event={"ID":"f67b152b-5cc3-4ae2-bcfb-6c16f511a340","Type":"ContainerStarted","Data":"a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749"} Sep 10 19:47:42.591855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:42.591482 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" event={"ID":"f67b152b-5cc3-4ae2-bcfb-6c16f511a340","Type":"ContainerStarted","Data":"e077076262ea85e1e8c9ccce3c122741a07fb81938ecd5142b96f2b056fbaf0b"} Sep 10 19:47:42.612363 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:42.612302 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" podStartSLOduration=1.612273874 podStartE2EDuration="1.612273874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:47:42.610138506 +0000 UTC m=+1648.577950755" watchObservedRunningTime="2026-09-10 19:47:42.612273874 +0000 UTC m=+1648.580086170" Sep 10 19:47:51.525156 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:51.525106 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:51.525815 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:51.525174 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:51.537698 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:51.537667 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:47:51.648463 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:47:51.648421 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:48:05.684251 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:05.684220 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq_11c51f02-21c3-481f-b360-374a0f9d6973/payload-processing/0.log" Sep 10 19:48:12.649071 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.649027 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs"] Sep 10 19:48:12.652756 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.652730 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:12.662250 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.662230 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-855446\"" Sep 10 19:48:12.662409 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.662385 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-855446-dockercfg-9f5f6\"" Sep 10 19:48:12.698316 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.698244 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs"] Sep 10 19:48:12.700816 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.700790 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx"] Sep 10 19:48:12.704430 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.704411 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:12.712603 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.712561 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9gmv\" (UniqueName: \"kubernetes.io/projected/b2f0b2e2-d17e-4484-af36-48d799404c24-kube-api-access-k9gmv\") pod \"payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs\" (UID: \"b2f0b2e2-d17e-4484-af36-48d799404c24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:12.712932 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.712900 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f0b2e2-d17e-4484-af36-48d799404c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs\" (UID: \"b2f0b2e2-d17e-4484-af36-48d799404c24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:12.726340 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.726302 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx"] Sep 10 19:48:12.814379 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.814340 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f0b2e2-d17e-4484-af36-48d799404c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs\" (UID: \"b2f0b2e2-d17e-4484-af36-48d799404c24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:12.814606 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.814428 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47eadba4-bc9b-4c1f-b964-99d9024caece-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx\" (UID: \"47eadba4-bc9b-4c1f-b964-99d9024caece\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:12.814606 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.814474 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpgq\" (UniqueName: \"kubernetes.io/projected/47eadba4-bc9b-4c1f-b964-99d9024caece-kube-api-access-8lpgq\") pod \"payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx\" (UID: \"47eadba4-bc9b-4c1f-b964-99d9024caece\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:12.814606 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.814550 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9gmv\" (UniqueName: \"kubernetes.io/projected/b2f0b2e2-d17e-4484-af36-48d799404c24-kube-api-access-k9gmv\") pod \"payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs\" (UID: \"b2f0b2e2-d17e-4484-af36-48d799404c24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:12.815094 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.815072 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f0b2e2-d17e-4484-af36-48d799404c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs\" (UID: \"b2f0b2e2-d17e-4484-af36-48d799404c24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:12.834899 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.834859 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9gmv\" (UniqueName: \"kubernetes.io/projected/b2f0b2e2-d17e-4484-af36-48d799404c24-kube-api-access-k9gmv\") pod \"payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs\" (UID: \"b2f0b2e2-d17e-4484-af36-48d799404c24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:12.915465 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.915345 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47eadba4-bc9b-4c1f-b964-99d9024caece-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx\" (UID: \"47eadba4-bc9b-4c1f-b964-99d9024caece\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:12.915465 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.915424 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpgq\" (UniqueName: \"kubernetes.io/projected/47eadba4-bc9b-4c1f-b964-99d9024caece-kube-api-access-8lpgq\") pod \"payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx\" (UID: \"47eadba4-bc9b-4c1f-b964-99d9024caece\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:12.916463 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.916436 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47eadba4-bc9b-4c1f-b964-99d9024caece-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx\" (UID: \"47eadba4-bc9b-4c1f-b964-99d9024caece\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:12.926625 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.926598 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpgq\" (UniqueName: \"kubernetes.io/projected/47eadba4-bc9b-4c1f-b964-99d9024caece-kube-api-access-8lpgq\") pod \"payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx\" (UID: \"47eadba4-bc9b-4c1f-b964-99d9024caece\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:12.964624 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:12.964587 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:13.019759 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.019667 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:13.249338 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.249311 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs"] Sep 10 19:48:13.251403 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:48:13.251363 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f0b2e2_d17e_4484_af36_48d799404c24.slice/crio-c0322c0538cb0c62e01b9bcf6b381b8d6cea267447813da42ce25f0e69383a39 WatchSource:0}: Error finding container c0322c0538cb0c62e01b9bcf6b381b8d6cea267447813da42ce25f0e69383a39: Status 404 returned error can't find the container with id c0322c0538cb0c62e01b9bcf6b381b8d6cea267447813da42ce25f0e69383a39 Sep 10 19:48:13.326607 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.326572 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx"] Sep 10 19:48:13.329789 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:48:13.329747 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47eadba4_bc9b_4c1f_b964_99d9024caece.slice/crio-a25bf7fa96b8dc7bba8ae554cd826af895dc159c36466781b08d2cac5a6a2670 WatchSource:0}: Error finding container a25bf7fa96b8dc7bba8ae554cd826af895dc159c36466781b08d2cac5a6a2670: Status 404 returned error can't find the container with id a25bf7fa96b8dc7bba8ae554cd826af895dc159c36466781b08d2cac5a6a2670 Sep 10 19:48:13.759606 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.759473 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" event={"ID":"b2f0b2e2-d17e-4484-af36-48d799404c24","Type":"ContainerStarted","Data":"47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a"} Sep 10 19:48:13.759606 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.759543 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" event={"ID":"b2f0b2e2-d17e-4484-af36-48d799404c24","Type":"ContainerStarted","Data":"c0322c0538cb0c62e01b9bcf6b381b8d6cea267447813da42ce25f0e69383a39"} Sep 10 19:48:13.759606 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.759574 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:13.761461 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.761430 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" event={"ID":"47eadba4-bc9b-4c1f-b964-99d9024caece","Type":"ContainerStarted","Data":"9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a"} Sep 10 19:48:13.761601 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.761484 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" event={"ID":"47eadba4-bc9b-4c1f-b964-99d9024caece","Type":"ContainerStarted","Data":"a25bf7fa96b8dc7bba8ae554cd826af895dc159c36466781b08d2cac5a6a2670"} Sep 10 19:48:13.761601 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.761537 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:13.788316 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.788248 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" podStartSLOduration=1.788234164 podStartE2EDuration="1.788234164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:48:13.785703234 +0000 UTC m=+1679.753515480" watchObservedRunningTime="2026-09-10 19:48:13.788234164 +0000 UTC m=+1679.756046411" Sep 10 19:48:13.817626 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:13.817567 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" podStartSLOduration=1.817546621 podStartE2EDuration="1.817546621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:48:13.816237555 +0000 UTC m=+1679.784049802" watchObservedRunningTime="2026-09-10 19:48:13.817546621 +0000 UTC m=+1679.785358868" Sep 10 19:48:24.768484 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:24.768449 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:24.769031 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:24.768676 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:52.548446 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.548400 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx"] Sep 10 19:48:52.548908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.548674 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" podUID="47eadba4-bc9b-4c1f-b964-99d9024caece" containerName="payload-processing" containerID="cri-o://9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a" gracePeriod=30 Sep 10 19:48:52.558942 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.558910 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs"] Sep 10 19:48:52.559119 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.559098 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" podUID="b2f0b2e2-d17e-4484-af36-48d799404c24" containerName="payload-pre-processing" containerID="cri-o://47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a" gracePeriod=30 Sep 10 19:48:52.854625 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.854595 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:52.857610 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.857586 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:52.976253 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.976218 2584 generic.go:356] "Generic (PLEG): container finished" podID="47eadba4-bc9b-4c1f-b964-99d9024caece" containerID="9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a" exitCode=0 Sep 10 19:48:52.976453 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.976267 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" Sep 10 19:48:52.976453 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.976311 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" event={"ID":"47eadba4-bc9b-4c1f-b964-99d9024caece","Type":"ContainerDied","Data":"9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a"} Sep 10 19:48:52.976453 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.976368 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx" event={"ID":"47eadba4-bc9b-4c1f-b964-99d9024caece","Type":"ContainerDied","Data":"a25bf7fa96b8dc7bba8ae554cd826af895dc159c36466781b08d2cac5a6a2670"} Sep 10 19:48:52.976453 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.976400 2584 scope.go:117] "RemoveContainer" containerID="9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a" Sep 10 19:48:52.977669 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.977649 2584 generic.go:356] "Generic (PLEG): container finished" podID="b2f0b2e2-d17e-4484-af36-48d799404c24" containerID="47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a" exitCode=0 Sep 10 19:48:52.977771 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.977695 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" Sep 10 19:48:52.977771 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.977732 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" event={"ID":"b2f0b2e2-d17e-4484-af36-48d799404c24","Type":"ContainerDied","Data":"47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a"} Sep 10 19:48:52.977872 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.977787 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs" event={"ID":"b2f0b2e2-d17e-4484-af36-48d799404c24","Type":"ContainerDied","Data":"c0322c0538cb0c62e01b9bcf6b381b8d6cea267447813da42ce25f0e69383a39"} Sep 10 19:48:52.985316 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.985289 2584 scope.go:117] "RemoveContainer" containerID="9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a" Sep 10 19:48:52.985608 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:48:52.985585 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a\": container with ID starting with 9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a not found: ID does not exist" containerID="9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a" Sep 10 19:48:52.985680 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.985619 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a"} err="failed to get container status \"9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a\": rpc error: code = NotFound desc = could not find container \"9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a\": container with ID starting with 9b1c57da4123834ae49e18d6f586f0acb1dfe14f2a183f6cacb3ab5bf2c8397a not found: ID does not exist" Sep 10 19:48:52.985680 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.985636 2584 scope.go:117] "RemoveContainer" containerID="47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a" Sep 10 19:48:52.996161 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.996137 2584 scope.go:117] "RemoveContainer" containerID="47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a" Sep 10 19:48:52.996762 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:48:52.996728 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a\": container with ID starting with 47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a not found: ID does not exist" containerID="47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a" Sep 10 19:48:52.996869 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:52.996773 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a"} err="failed to get container status \"47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a\": rpc error: code = NotFound desc = could not find container \"47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a\": container with ID starting with 47f6c50c15d3e4441c667f61b7a2f88fd5dba0b65734108d852f1f4074a27c8a not found: ID does not exist" Sep 10 19:48:53.001120 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.001097 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47eadba4-bc9b-4c1f-b964-99d9024caece-plugins-config-volume\") pod \"47eadba4-bc9b-4c1f-b964-99d9024caece\" (UID: \"47eadba4-bc9b-4c1f-b964-99d9024caece\") " Sep 10 19:48:53.001225 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.001129 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lpgq\" (UniqueName: \"kubernetes.io/projected/47eadba4-bc9b-4c1f-b964-99d9024caece-kube-api-access-8lpgq\") pod \"47eadba4-bc9b-4c1f-b964-99d9024caece\" (UID: \"47eadba4-bc9b-4c1f-b964-99d9024caece\") " Sep 10 19:48:53.001225 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.001189 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f0b2e2-d17e-4484-af36-48d799404c24-plugins-config-volume\") pod \"b2f0b2e2-d17e-4484-af36-48d799404c24\" (UID: \"b2f0b2e2-d17e-4484-af36-48d799404c24\") " Sep 10 19:48:53.001454 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.001227 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9gmv\" (UniqueName: \"kubernetes.io/projected/b2f0b2e2-d17e-4484-af36-48d799404c24-kube-api-access-k9gmv\") pod \"b2f0b2e2-d17e-4484-af36-48d799404c24\" (UID: \"b2f0b2e2-d17e-4484-af36-48d799404c24\") " Sep 10 19:48:53.001600 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.001575 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47eadba4-bc9b-4c1f-b964-99d9024caece-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47eadba4-bc9b-4c1f-b964-99d9024caece" (UID: "47eadba4-bc9b-4c1f-b964-99d9024caece"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:48:53.001662 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.001621 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f0b2e2-d17e-4484-af36-48d799404c24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2f0b2e2-d17e-4484-af36-48d799404c24" (UID: "b2f0b2e2-d17e-4484-af36-48d799404c24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:48:53.003337 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.003316 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f0b2e2-d17e-4484-af36-48d799404c24-kube-api-access-k9gmv" (OuterVolumeSpecName: "kube-api-access-k9gmv") pod "b2f0b2e2-d17e-4484-af36-48d799404c24" (UID: "b2f0b2e2-d17e-4484-af36-48d799404c24"). InnerVolumeSpecName "kube-api-access-k9gmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:48:53.003397 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.003371 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47eadba4-bc9b-4c1f-b964-99d9024caece-kube-api-access-8lpgq" (OuterVolumeSpecName: "kube-api-access-8lpgq") pod "47eadba4-bc9b-4c1f-b964-99d9024caece" (UID: "47eadba4-bc9b-4c1f-b964-99d9024caece"). InnerVolumeSpecName "kube-api-access-8lpgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:48:53.102598 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.102500 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47eadba4-bc9b-4c1f-b964-99d9024caece-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:48:53.102598 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.102538 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpgq\" (UniqueName: \"kubernetes.io/projected/47eadba4-bc9b-4c1f-b964-99d9024caece-kube-api-access-8lpgq\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:48:53.102598 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.102548 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f0b2e2-d17e-4484-af36-48d799404c24-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:48:53.102598 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.102556 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9gmv\" (UniqueName: \"kubernetes.io/projected/b2f0b2e2-d17e-4484-af36-48d799404c24-kube-api-access-k9gmv\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:48:53.333789 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.333751 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs"] Sep 10 19:48:53.335609 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.335585 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-855446-75458bcf4b-fd4cs"] Sep 10 19:48:53.360802 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.360705 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx"] Sep 10 19:48:53.371908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:53.371875 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-855446-77df5cb767-hpvrx"] Sep 10 19:48:54.488875 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:54.488836 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47eadba4-bc9b-4c1f-b964-99d9024caece" path="/var/lib/kubelet/pods/47eadba4-bc9b-4c1f-b964-99d9024caece/volumes" Sep 10 19:48:54.489239 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:48:54.489198 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f0b2e2-d17e-4484-af36-48d799404c24" path="/var/lib/kubelet/pods/b2f0b2e2-d17e-4484-af36-48d799404c24/volumes" Sep 10 19:49:03.386572 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.386529 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn"] Sep 10 19:49:03.387007 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.386927 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47eadba4-bc9b-4c1f-b964-99d9024caece" containerName="payload-processing" Sep 10 19:49:03.387007 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.386938 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="47eadba4-bc9b-4c1f-b964-99d9024caece" containerName="payload-processing" Sep 10 19:49:03.387007 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.386951 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f0b2e2-d17e-4484-af36-48d799404c24" containerName="payload-pre-processing" Sep 10 19:49:03.387007 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.386957 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f0b2e2-d17e-4484-af36-48d799404c24" containerName="payload-pre-processing" Sep 10 19:49:03.387154 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.387018 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="47eadba4-bc9b-4c1f-b964-99d9024caece" containerName="payload-processing" Sep 10 19:49:03.387154 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.387029 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f0b2e2-d17e-4484-af36-48d799404c24" containerName="payload-pre-processing" Sep 10 19:49:03.391888 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.391864 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn" Sep 10 19:49:03.394612 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.394586 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zztzd\"" Sep 10 19:49:03.394768 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.394686 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:49:03.399152 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.399121 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn"] Sep 10 19:49:03.497120 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.497078 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39f24dc-8df6-4b0b-be78-d4a30ca96854-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn\" (UID: \"d39f24dc-8df6-4b0b-be78-d4a30ca96854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn" Sep 10 19:49:03.597986 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.597934 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39f24dc-8df6-4b0b-be78-d4a30ca96854-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn\" (UID: \"d39f24dc-8df6-4b0b-be78-d4a30ca96854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn" Sep 10 19:49:03.598640 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.598619 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39f24dc-8df6-4b0b-be78-d4a30ca96854-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn\" (UID: \"d39f24dc-8df6-4b0b-be78-d4a30ca96854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn" Sep 10 19:49:03.705142 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.705022 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn" Sep 10 19:49:03.981945 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:03.981917 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn"] Sep 10 19:49:03.983931 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:49:03.983886 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39f24dc_8df6_4b0b_be78_d4a30ca96854.slice/crio-2a9ca817c7b55211809965a9e8ae839414e7f5574ac9a39319146474e380ecc8 WatchSource:0}: Error finding container 2a9ca817c7b55211809965a9e8ae839414e7f5574ac9a39319146474e380ecc8: Status 404 returned error can't find the container with id 2a9ca817c7b55211809965a9e8ae839414e7f5574ac9a39319146474e380ecc8 Sep 10 19:49:04.038812 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:04.038777 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn" event={"ID":"d39f24dc-8df6-4b0b-be78-d4a30ca96854","Type":"ContainerStarted","Data":"2a9ca817c7b55211809965a9e8ae839414e7f5574ac9a39319146474e380ecc8"} Sep 10 19:49:05.045521 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:05.045421 2584 generic.go:356] "Generic (PLEG): container finished" podID="d39f24dc-8df6-4b0b-be78-d4a30ca96854" containerID="aee47c3429d43449a57b86246dd476244d7d6c7d1c2d1864a9f82aeab2d83fda" exitCode=0 Sep 10 19:49:05.045908 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:05.045515 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn" event={"ID":"d39f24dc-8df6-4b0b-be78-d4a30ca96854","Type":"ContainerDied","Data":"aee47c3429d43449a57b86246dd476244d7d6c7d1c2d1864a9f82aeab2d83fda"} Sep 10 19:49:06.188945 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:06.188914 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn" Sep 10 19:49:06.224960 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:06.224926 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39f24dc-8df6-4b0b-be78-d4a30ca96854-maas-api-service-ca\") pod \"d39f24dc-8df6-4b0b-be78-d4a30ca96854\" (UID: \"d39f24dc-8df6-4b0b-be78-d4a30ca96854\") " Sep 10 19:49:06.225401 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:06.225375 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39f24dc-8df6-4b0b-be78-d4a30ca96854-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d39f24dc-8df6-4b0b-be78-d4a30ca96854" (UID: "d39f24dc-8df6-4b0b-be78-d4a30ca96854"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:49:06.326100 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:06.326056 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39f24dc-8df6-4b0b-be78-d4a30ca96854-maas-api-service-ca\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:49:07.057348 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:07.057312 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn" event={"ID":"d39f24dc-8df6-4b0b-be78-d4a30ca96854","Type":"ContainerDied","Data":"2a9ca817c7b55211809965a9e8ae839414e7f5574ac9a39319146474e380ecc8"} Sep 10 19:49:07.057501 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:07.057372 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a9ca817c7b55211809965a9e8ae839414e7f5574ac9a39319146474e380ecc8" Sep 10 19:49:07.057501 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:07.057323 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn" Sep 10 19:49:19.495519 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:19.495478 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn"] Sep 10 19:49:19.501596 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:19.501564 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-836291-x7qkn"] Sep 10 19:49:20.489103 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:20.489067 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d39f24dc-8df6-4b0b-be78-d4a30ca96854" path="/var/lib/kubelet/pods/d39f24dc-8df6-4b0b-be78-d4a30ca96854/volumes" Sep 10 19:49:24.845138 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:24.845096 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8"] Sep 10 19:49:39.865687 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:39.865618 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" podUID="f67b152b-5cc3-4ae2-bcfb-6c16f511a340" containerName="main" containerID="cri-o://a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749" gracePeriod=45 Sep 10 19:49:40.123678 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.123610 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:49:40.231852 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.231809 2584 generic.go:356] "Generic (PLEG): container finished" podID="f67b152b-5cc3-4ae2-bcfb-6c16f511a340" containerID="a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749" exitCode=0 Sep 10 19:49:40.231852 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.231852 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" Sep 10 19:49:40.232118 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.231905 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" event={"ID":"f67b152b-5cc3-4ae2-bcfb-6c16f511a340","Type":"ContainerDied","Data":"a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749"} Sep 10 19:49:40.232118 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.231958 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8" event={"ID":"f67b152b-5cc3-4ae2-bcfb-6c16f511a340","Type":"ContainerDied","Data":"e077076262ea85e1e8c9ccce3c122741a07fb81938ecd5142b96f2b056fbaf0b"} Sep 10 19:49:40.232118 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.231987 2584 scope.go:117] "RemoveContainer" containerID="a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749" Sep 10 19:49:40.241044 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.241023 2584 scope.go:117] "RemoveContainer" containerID="a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749" Sep 10 19:49:40.241405 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:49:40.241379 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749\": container with ID starting with a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749 not found: ID does not exist" containerID="a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749" Sep 10 19:49:40.241477 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.241420 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749"} err="failed to get container status \"a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749\": rpc error: code = NotFound desc = could not find container \"a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749\": container with ID starting with a763fc1f43cfd6b7ccb5b84ee9c14e106e11062e66dc1a6d4802fdf0b6704749 not found: ID does not exist" Sep 10 19:49:40.258768 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.258737 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tmp-dir\") pod \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " Sep 10 19:49:40.258905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.258783 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9q6c\" (UniqueName: \"kubernetes.io/projected/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-kube-api-access-g9q6c\") pod \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " Sep 10 19:49:40.258905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.258811 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-home\") pod \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " Sep 10 19:49:40.258905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.258837 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-dshm\") pod \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " Sep 10 19:49:40.258905 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.258888 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tls-certs\") pod \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " Sep 10 19:49:40.259084 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.259066 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f67b152b-5cc3-4ae2-bcfb-6c16f511a340" (UID: "f67b152b-5cc3-4ae2-bcfb-6c16f511a340"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:49:40.259141 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.259076 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-model-cache\") pod \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\" (UID: \"f67b152b-5cc3-4ae2-bcfb-6c16f511a340\") " Sep 10 19:49:40.259141 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.259123 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-home" (OuterVolumeSpecName: "home") pod "f67b152b-5cc3-4ae2-bcfb-6c16f511a340" (UID: "f67b152b-5cc3-4ae2-bcfb-6c16f511a340"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:49:40.259379 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.259356 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-model-cache" (OuterVolumeSpecName: "model-cache") pod "f67b152b-5cc3-4ae2-bcfb-6c16f511a340" (UID: "f67b152b-5cc3-4ae2-bcfb-6c16f511a340"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:49:40.259520 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.259499 2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-model-cache\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:49:40.259520 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.259521 2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tmp-dir\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:49:40.259663 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.259533 2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-home\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:49:40.261111 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.261088 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f67b152b-5cc3-4ae2-bcfb-6c16f511a340" (UID: "f67b152b-5cc3-4ae2-bcfb-6c16f511a340"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:49:40.261480 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.261452 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-dshm" (OuterVolumeSpecName: "dshm") pod "f67b152b-5cc3-4ae2-bcfb-6c16f511a340" (UID: "f67b152b-5cc3-4ae2-bcfb-6c16f511a340"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:49:40.261566 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.261468 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-kube-api-access-g9q6c" (OuterVolumeSpecName: "kube-api-access-g9q6c") pod "f67b152b-5cc3-4ae2-bcfb-6c16f511a340" (UID: "f67b152b-5cc3-4ae2-bcfb-6c16f511a340"). InnerVolumeSpecName "kube-api-access-g9q6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:49:40.360363 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.360319 2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-tls-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:49:40.360363 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.360357 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9q6c\" (UniqueName: \"kubernetes.io/projected/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-kube-api-access-g9q6c\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:49:40.360363 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.360367 2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67b152b-5cc3-4ae2-bcfb-6c16f511a340-dshm\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:49:40.549319 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.549264 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8"] Sep 10 19:49:40.553952 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:40.553914 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-836291/ipp-route-836291-kserve-86c8785675-vj2k8"] Sep 10 19:49:42.489382 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:49:42.489345 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67b152b-5cc3-4ae2-bcfb-6c16f511a340" path="/var/lib/kubelet/pods/f67b152b-5cc3-4ae2-bcfb-6c16f511a340/volumes" Sep 10 19:50:23.566397 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.566357 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx"] Sep 10 19:50:23.566912 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.566894 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d39f24dc-8df6-4b0b-be78-d4a30ca96854" containerName="revoke-keys" Sep 10 19:50:23.566958 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.566914 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d39f24dc-8df6-4b0b-be78-d4a30ca96854" containerName="revoke-keys" Sep 10 19:50:23.566958 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.566923 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67b152b-5cc3-4ae2-bcfb-6c16f511a340" containerName="main" Sep 10 19:50:23.566958 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.566929 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67b152b-5cc3-4ae2-bcfb-6c16f511a340" containerName="main" Sep 10 19:50:23.567056 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.566986 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d39f24dc-8df6-4b0b-be78-d4a30ca96854" containerName="revoke-keys" Sep 10 19:50:23.567056 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.566997 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67b152b-5cc3-4ae2-bcfb-6c16f511a340" containerName="main" Sep 10 19:50:23.570104 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.570086 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" Sep 10 19:50:23.572919 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.572894 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:50:23.573014 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.573002 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zztzd\"" Sep 10 19:50:23.582473 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.582444 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx"] Sep 10 19:50:23.667350 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.667312 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/447183c2-e539-4e46-98de-483fad964448-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx\" (UID: \"447183c2-e539-4e46-98de-483fad964448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" Sep 10 19:50:23.768146 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.768110 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/447183c2-e539-4e46-98de-483fad964448-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx\" (UID: \"447183c2-e539-4e46-98de-483fad964448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" Sep 10 19:50:23.768804 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.768783 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/447183c2-e539-4e46-98de-483fad964448-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx\" (UID: \"447183c2-e539-4e46-98de-483fad964448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" Sep 10 19:50:23.881612 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:23.881510 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" Sep 10 19:50:24.159679 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:24.159648 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx"] Sep 10 19:50:24.161838 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:50:24.161806 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447183c2_e539_4e46_98de_483fad964448.slice/crio-17fdb7daa3f979b7aa9ab13e94e3600dc3c474cfadaafca534d051bf78b7fde2 WatchSource:0}: Error finding container 17fdb7daa3f979b7aa9ab13e94e3600dc3c474cfadaafca534d051bf78b7fde2: Status 404 returned error can't find the container with id 17fdb7daa3f979b7aa9ab13e94e3600dc3c474cfadaafca534d051bf78b7fde2 Sep 10 19:50:24.459029 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:24.458987 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" event={"ID":"447183c2-e539-4e46-98de-483fad964448","Type":"ContainerStarted","Data":"2751a6d36730c46dabe88c25a4dc44caf1f2a70ac5f5e21d2ea503039731d702"} Sep 10 19:50:24.459217 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:24.459044 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" event={"ID":"447183c2-e539-4e46-98de-483fad964448","Type":"ContainerStarted","Data":"17fdb7daa3f979b7aa9ab13e94e3600dc3c474cfadaafca534d051bf78b7fde2"} Sep 10 19:50:24.475922 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:24.475865 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" podStartSLOduration=1.475848062 podStartE2EDuration="1.475848062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:50:24.473545577 +0000 UTC m=+1810.441357825" watchObservedRunningTime="2026-09-10 19:50:24.475848062 +0000 UTC m=+1810.443660308" Sep 10 19:50:25.465588 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:25.465550 2584 generic.go:356] "Generic (PLEG): container finished" podID="447183c2-e539-4e46-98de-483fad964448" containerID="2751a6d36730c46dabe88c25a4dc44caf1f2a70ac5f5e21d2ea503039731d702" exitCode=0 Sep 10 19:50:25.465985 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:25.465637 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" event={"ID":"447183c2-e539-4e46-98de-483fad964448","Type":"ContainerDied","Data":"2751a6d36730c46dabe88c25a4dc44caf1f2a70ac5f5e21d2ea503039731d702"} Sep 10 19:50:26.608691 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:26.608655 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" Sep 10 19:50:26.696763 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:26.696719 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/447183c2-e539-4e46-98de-483fad964448-maas-api-service-ca\") pod \"447183c2-e539-4e46-98de-483fad964448\" (UID: \"447183c2-e539-4e46-98de-483fad964448\") " Sep 10 19:50:26.697093 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:26.697068 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447183c2-e539-4e46-98de-483fad964448-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "447183c2-e539-4e46-98de-483fad964448" (UID: "447183c2-e539-4e46-98de-483fad964448"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:50:26.798178 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:26.798086 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/447183c2-e539-4e46-98de-483fad964448-maas-api-service-ca\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:50:27.477567 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:27.477529 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" event={"ID":"447183c2-e539-4e46-98de-483fad964448","Type":"ContainerDied","Data":"17fdb7daa3f979b7aa9ab13e94e3600dc3c474cfadaafca534d051bf78b7fde2"} Sep 10 19:50:27.477752 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:27.477579 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17fdb7daa3f979b7aa9ab13e94e3600dc3c474cfadaafca534d051bf78b7fde2" Sep 10 19:50:27.477752 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:27.477539 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx" Sep 10 19:50:34.193260 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.193221 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq"] Sep 10 19:50:34.193744 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.193455 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" podUID="11c51f02-21c3-481f-b360-374a0f9d6973" containerName="payload-processing" containerID="cri-o://8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d" gracePeriod=30 Sep 10 19:50:34.202591 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.202563 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj"] Sep 10 19:50:34.202837 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.202779 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" podUID="b72c938c-ce6d-498e-b164-8ccff23c7df9" containerName="payload-pre-processing" containerID="cri-o://a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b" gracePeriod=30 Sep 10 19:50:34.402655 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.402620 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx"] Sep 10 19:50:34.404749 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.404700 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b588eb-5zlpx"] Sep 10 19:50:34.481474 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.481447 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:50:34.484498 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.484473 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:50:34.488793 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.488764 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="447183c2-e539-4e46-98de-483fad964448" path="/var/lib/kubelet/pods/447183c2-e539-4e46-98de-483fad964448/volumes" Sep 10 19:50:34.514999 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.514958 2584 generic.go:356] "Generic (PLEG): container finished" podID="b72c938c-ce6d-498e-b164-8ccff23c7df9" containerID="a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b" exitCode=0 Sep 10 19:50:34.515202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.515011 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" Sep 10 19:50:34.515202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.515010 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" event={"ID":"b72c938c-ce6d-498e-b164-8ccff23c7df9","Type":"ContainerDied","Data":"a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b"} Sep 10 19:50:34.515202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.515149 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj" event={"ID":"b72c938c-ce6d-498e-b164-8ccff23c7df9","Type":"ContainerDied","Data":"45ec6e7b901842365a2a33564f8c501616c4c4d0672e17c31c61798bd0dc7ace"} Sep 10 19:50:34.515202 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.515182 2584 scope.go:117] "RemoveContainer" containerID="a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b" Sep 10 19:50:34.516822 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.516797 2584 generic.go:356] "Generic (PLEG): container finished" podID="11c51f02-21c3-481f-b360-374a0f9d6973" containerID="8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d" exitCode=0 Sep 10 19:50:34.516931 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.516834 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" event={"ID":"11c51f02-21c3-481f-b360-374a0f9d6973","Type":"ContainerDied","Data":"8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d"} Sep 10 19:50:34.516931 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.516847 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" Sep 10 19:50:34.516931 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.516887 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq" event={"ID":"11c51f02-21c3-481f-b360-374a0f9d6973","Type":"ContainerDied","Data":"c61d33dbd734701b8578771f390b5a1ad9d5dce05bf87757757b838d60dc1b9a"} Sep 10 19:50:34.531039 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.531007 2584 scope.go:117] "RemoveContainer" containerID="a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b" Sep 10 19:50:34.541561 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:50:34.539506 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b\": container with ID starting with a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b not found: ID does not exist" containerID="a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b" Sep 10 19:50:34.541561 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.539558 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b"} err="failed to get container status \"a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b\": rpc error: code = NotFound desc = could not find container \"a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b\": container with ID starting with a960448b4fe1b45807b6a1aa5f7218a01594a0f9e294289b27e1db18ffc1608b not found: ID does not exist" Sep 10 19:50:34.541561 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.539586 2584 scope.go:117] "RemoveContainer" containerID="8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d" Sep 10 19:50:34.553297 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.553007 2584 scope.go:117] "RemoveContainer" containerID="8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d" Sep 10 19:50:34.554374 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:50:34.554047 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d\": container with ID starting with 8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d not found: ID does not exist" containerID="8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d" Sep 10 19:50:34.554374 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.554094 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d"} err="failed to get container status \"8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d\": rpc error: code = NotFound desc = could not find container \"8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d\": container with ID starting with 8f078164c80825820fdff42e882ae04b446f5bdb4d84f8ccfc91c93ab5215a8d not found: ID does not exist" Sep 10 19:50:34.575369 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.575321 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bpm8\" (UniqueName: \"kubernetes.io/projected/11c51f02-21c3-481f-b360-374a0f9d6973-kube-api-access-4bpm8\") pod \"11c51f02-21c3-481f-b360-374a0f9d6973\" (UID: \"11c51f02-21c3-481f-b360-374a0f9d6973\") " Sep 10 19:50:34.575557 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.575399 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrwd9\" (UniqueName: \"kubernetes.io/projected/b72c938c-ce6d-498e-b164-8ccff23c7df9-kube-api-access-zrwd9\") pod \"b72c938c-ce6d-498e-b164-8ccff23c7df9\" (UID: \"b72c938c-ce6d-498e-b164-8ccff23c7df9\") " Sep 10 19:50:34.575557 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.575492 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c51f02-21c3-481f-b360-374a0f9d6973-plugins-config-volume\") pod \"11c51f02-21c3-481f-b360-374a0f9d6973\" (UID: \"11c51f02-21c3-481f-b360-374a0f9d6973\") " Sep 10 19:50:34.575674 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.575583 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72c938c-ce6d-498e-b164-8ccff23c7df9-plugins-config-volume\") pod \"b72c938c-ce6d-498e-b164-8ccff23c7df9\" (UID: \"b72c938c-ce6d-498e-b164-8ccff23c7df9\") " Sep 10 19:50:34.576007 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.575979 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c51f02-21c3-481f-b360-374a0f9d6973-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11c51f02-21c3-481f-b360-374a0f9d6973" (UID: "11c51f02-21c3-481f-b360-374a0f9d6973"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:50:34.576083 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.576025 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72c938c-ce6d-498e-b164-8ccff23c7df9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b72c938c-ce6d-498e-b164-8ccff23c7df9" (UID: "b72c938c-ce6d-498e-b164-8ccff23c7df9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:50:34.577835 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.577802 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c51f02-21c3-481f-b360-374a0f9d6973-kube-api-access-4bpm8" (OuterVolumeSpecName: "kube-api-access-4bpm8") pod "11c51f02-21c3-481f-b360-374a0f9d6973" (UID: "11c51f02-21c3-481f-b360-374a0f9d6973"). InnerVolumeSpecName "kube-api-access-4bpm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:50:34.577835 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.577809 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72c938c-ce6d-498e-b164-8ccff23c7df9-kube-api-access-zrwd9" (OuterVolumeSpecName: "kube-api-access-zrwd9") pod "b72c938c-ce6d-498e-b164-8ccff23c7df9" (UID: "b72c938c-ce6d-498e-b164-8ccff23c7df9"). InnerVolumeSpecName "kube-api-access-zrwd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:50:34.677496 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.677454 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72c938c-ce6d-498e-b164-8ccff23c7df9-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:50:34.677496 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.677488 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bpm8\" (UniqueName: \"kubernetes.io/projected/11c51f02-21c3-481f-b360-374a0f9d6973-kube-api-access-4bpm8\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:50:34.677496 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.677497 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrwd9\" (UniqueName: \"kubernetes.io/projected/b72c938c-ce6d-498e-b164-8ccff23c7df9-kube-api-access-zrwd9\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:50:34.677496 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.677506 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c51f02-21c3-481f-b360-374a0f9d6973-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:50:34.863106 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.863068 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj"] Sep 10 19:50:34.870326 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.870292 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b588eb-6c67584d78-x5rqj"] Sep 10 19:50:34.894396 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.894358 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq"] Sep 10 19:50:34.901945 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:34.901913 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b588eb-cfcdd5c7f-6vgkq"] Sep 10 19:50:36.488787 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:36.488751 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11c51f02-21c3-481f-b360-374a0f9d6973" path="/var/lib/kubelet/pods/11c51f02-21c3-481f-b360-374a0f9d6973/volumes" Sep 10 19:50:36.489304 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:36.489264 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b72c938c-ce6d-498e-b164-8ccff23c7df9" path="/var/lib/kubelet/pods/b72c938c-ce6d-498e-b164-8ccff23c7df9/volumes" Sep 10 19:50:41.824471 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.824430 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2"] Sep 10 19:50:41.824859 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.824833 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c51f02-21c3-481f-b360-374a0f9d6973" containerName="payload-processing" Sep 10 19:50:41.824859 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.824846 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c51f02-21c3-481f-b360-374a0f9d6973" containerName="payload-processing" Sep 10 19:50:41.824955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.824865 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b72c938c-ce6d-498e-b164-8ccff23c7df9" containerName="payload-pre-processing" Sep 10 19:50:41.824955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.824873 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b72c938c-ce6d-498e-b164-8ccff23c7df9" containerName="payload-pre-processing" Sep 10 19:50:41.824955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.824882 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447183c2-e539-4e46-98de-483fad964448" containerName="revoke-keys" Sep 10 19:50:41.824955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.824890 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="447183c2-e539-4e46-98de-483fad964448" containerName="revoke-keys" Sep 10 19:50:41.824955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.824944 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="447183c2-e539-4e46-98de-483fad964448" containerName="revoke-keys" Sep 10 19:50:41.824955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.824951 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b72c938c-ce6d-498e-b164-8ccff23c7df9" containerName="payload-pre-processing" Sep 10 19:50:41.825133 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.824958 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c51f02-21c3-481f-b360-374a0f9d6973" containerName="payload-processing" Sep 10 19:50:41.829622 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.829600 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2" Sep 10 19:50:41.832731 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.832683 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:50:41.832914 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.832815 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zztzd\"" Sep 10 19:50:41.850462 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.850429 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2"] Sep 10 19:50:41.949203 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:41.949161 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7577377e-99bc-4b49-9de6-0596d8b4a367-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2\" (UID: \"7577377e-99bc-4b49-9de6-0596d8b4a367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2" Sep 10 19:50:42.050183 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:42.050139 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7577377e-99bc-4b49-9de6-0596d8b4a367-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2\" (UID: \"7577377e-99bc-4b49-9de6-0596d8b4a367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2" Sep 10 19:50:42.050832 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:42.050811 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7577377e-99bc-4b49-9de6-0596d8b4a367-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2\" (UID: \"7577377e-99bc-4b49-9de6-0596d8b4a367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2" Sep 10 19:50:42.139477 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:42.139376 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2" Sep 10 19:50:42.419568 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:42.419542 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2"] Sep 10 19:50:42.421824 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:50:42.421788 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7577377e_99bc_4b49_9de6_0596d8b4a367.slice/crio-813e73f36e56b7c31f7926d84a42d16a134f3eb6f6b41bcd46367b1fc8d57eaf WatchSource:0}: Error finding container 813e73f36e56b7c31f7926d84a42d16a134f3eb6f6b41bcd46367b1fc8d57eaf: Status 404 returned error can't find the container with id 813e73f36e56b7c31f7926d84a42d16a134f3eb6f6b41bcd46367b1fc8d57eaf Sep 10 19:50:42.566978 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:42.566939 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2" event={"ID":"7577377e-99bc-4b49-9de6-0596d8b4a367","Type":"ContainerStarted","Data":"813e73f36e56b7c31f7926d84a42d16a134f3eb6f6b41bcd46367b1fc8d57eaf"} Sep 10 19:50:43.573809 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:43.573774 2584 generic.go:356] "Generic (PLEG): container finished" podID="7577377e-99bc-4b49-9de6-0596d8b4a367" containerID="ad1de8c5602e1704f4d8e57c353b7621f37bc957b52e152195c3c1bf364110b1" exitCode=0 Sep 10 19:50:43.574174 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:43.573851 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2" event={"ID":"7577377e-99bc-4b49-9de6-0596d8b4a367","Type":"ContainerDied","Data":"ad1de8c5602e1704f4d8e57c353b7621f37bc957b52e152195c3c1bf364110b1"} Sep 10 19:50:44.721016 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:44.720980 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2" Sep 10 19:50:44.878224 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:44.878130 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7577377e-99bc-4b49-9de6-0596d8b4a367-maas-api-service-ca\") pod \"7577377e-99bc-4b49-9de6-0596d8b4a367\" (UID: \"7577377e-99bc-4b49-9de6-0596d8b4a367\") " Sep 10 19:50:44.878536 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:44.878514 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7577377e-99bc-4b49-9de6-0596d8b4a367-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7577377e-99bc-4b49-9de6-0596d8b4a367" (UID: "7577377e-99bc-4b49-9de6-0596d8b4a367"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:50:44.979345 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:44.979306 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7577377e-99bc-4b49-9de6-0596d8b4a367-maas-api-service-ca\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:50:45.588007 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:45.587975 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2" Sep 10 19:50:45.588193 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:45.587980 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2" event={"ID":"7577377e-99bc-4b49-9de6-0596d8b4a367","Type":"ContainerDied","Data":"813e73f36e56b7c31f7926d84a42d16a134f3eb6f6b41bcd46367b1fc8d57eaf"} Sep 10 19:50:45.588193 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:45.588105 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="813e73f36e56b7c31f7926d84a42d16a134f3eb6f6b41bcd46367b1fc8d57eaf" Sep 10 19:50:52.587573 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:52.587535 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2"] Sep 10 19:50:52.589875 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:52.589321 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-07c5fe-r4gt2"] Sep 10 19:50:54.487601 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:54.487563 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7577377e-99bc-4b49-9de6-0596d8b4a367" path="/var/lib/kubelet/pods/7577377e-99bc-4b49-9de6-0596d8b4a367/volumes" Sep 10 19:50:59.700173 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:59.700134 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp"] Sep 10 19:50:59.700636 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:50:59.700348 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" containerID="cri-o://f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f" gracePeriod=30 Sep 10 19:51:02.070330 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:02.070270 2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 19:51:02.070718 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:02.070349 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 19:51:04.959966 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:04.959939 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:51:05.064775 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.064733 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-podinfo\") pod \"361212c6-d3d6-454f-ae24-f2882b1e3469\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " Sep 10 19:51:05.064775 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.064780 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-envoy\") pod \"361212c6-d3d6-454f-ae24-f2882b1e3469\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " Sep 10 19:51:05.065042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.064801 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-credential-socket\") pod \"361212c6-d3d6-454f-ae24-f2882b1e3469\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " Sep 10 19:51:05.065042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.064819 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-socket\") pod \"361212c6-d3d6-454f-ae24-f2882b1e3469\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " Sep 10 19:51:05.065042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.064844 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm7v7\" (UniqueName: \"kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-kube-api-access-bm7v7\") pod \"361212c6-d3d6-454f-ae24-f2882b1e3469\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " Sep 10 19:51:05.065042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.064890 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-certs\") pod \"361212c6-d3d6-454f-ae24-f2882b1e3469\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " Sep 10 19:51:05.065042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.064919 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361212c6-d3d6-454f-ae24-f2882b1e3469-istiod-ca-cert\") pod \"361212c6-d3d6-454f-ae24-f2882b1e3469\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " Sep 10 19:51:05.065042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.064947 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-data\") pod \"361212c6-d3d6-454f-ae24-f2882b1e3469\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " Sep 10 19:51:05.065042 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.064986 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-token\") pod \"361212c6-d3d6-454f-ae24-f2882b1e3469\" (UID: \"361212c6-d3d6-454f-ae24-f2882b1e3469\") " Sep 10 19:51:05.065428 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.065161 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "361212c6-d3d6-454f-ae24-f2882b1e3469" (UID: "361212c6-d3d6-454f-ae24-f2882b1e3469"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:51:05.065428 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.065389 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-data" (OuterVolumeSpecName: "istio-data") pod "361212c6-d3d6-454f-ae24-f2882b1e3469" (UID: "361212c6-d3d6-454f-ae24-f2882b1e3469"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:51:05.065428 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.065397 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-credential-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:05.065428 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.065169 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "361212c6-d3d6-454f-ae24-f2882b1e3469" (UID: "361212c6-d3d6-454f-ae24-f2882b1e3469"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:51:05.065645 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.065503 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "361212c6-d3d6-454f-ae24-f2882b1e3469" (UID: "361212c6-d3d6-454f-ae24-f2882b1e3469"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:51:05.065645 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.065528 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361212c6-d3d6-454f-ae24-f2882b1e3469-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "361212c6-d3d6-454f-ae24-f2882b1e3469" (UID: "361212c6-d3d6-454f-ae24-f2882b1e3469"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:51:05.067168 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.067144 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "361212c6-d3d6-454f-ae24-f2882b1e3469" (UID: "361212c6-d3d6-454f-ae24-f2882b1e3469"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:51:05.067309 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.067198 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "361212c6-d3d6-454f-ae24-f2882b1e3469" (UID: "361212c6-d3d6-454f-ae24-f2882b1e3469"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:51:05.067309 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.067203 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-token" (OuterVolumeSpecName: "istio-token") pod "361212c6-d3d6-454f-ae24-f2882b1e3469" (UID: "361212c6-d3d6-454f-ae24-f2882b1e3469"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:51:05.067407 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.067309 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-kube-api-access-bm7v7" (OuterVolumeSpecName: "kube-api-access-bm7v7") pod "361212c6-d3d6-454f-ae24-f2882b1e3469" (UID: "361212c6-d3d6-454f-ae24-f2882b1e3469"). InnerVolumeSpecName "kube-api-access-bm7v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:51:05.166791 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.166747 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-podinfo\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:05.166791 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.166783 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-envoy\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:05.166791 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.166794 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:05.166791 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.166802 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm7v7\" (UniqueName: \"kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-kube-api-access-bm7v7\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:05.167074 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.166812 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-workload-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:05.167074 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.166820 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361212c6-d3d6-454f-ae24-f2882b1e3469-istiod-ca-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:05.167074 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.166827 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-data\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:05.167074 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.166834 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361212c6-d3d6-454f-ae24-f2882b1e3469-istio-token\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:05.700568 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.700536 2584 generic.go:356] "Generic (PLEG): container finished" podID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerID="f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f" exitCode=0 Sep 10 19:51:05.700762 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.700582 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" Sep 10 19:51:05.700762 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.700613 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" event={"ID":"361212c6-d3d6-454f-ae24-f2882b1e3469","Type":"ContainerDied","Data":"f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f"} Sep 10 19:51:05.700762 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.700684 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp" event={"ID":"361212c6-d3d6-454f-ae24-f2882b1e3469","Type":"ContainerDied","Data":"f6e95a7ad27de3ca6636bb1684c0d08bd84a32df122936a0d53ce8b1fbab4bbf"} Sep 10 19:51:05.700762 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.700716 2584 scope.go:117] "RemoveContainer" containerID="f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f" Sep 10 19:51:05.710806 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.710781 2584 scope.go:117] "RemoveContainer" containerID="f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34" Sep 10 19:51:05.719363 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.719339 2584 scope.go:117] "RemoveContainer" containerID="f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f" Sep 10 19:51:05.719694 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:51:05.719674 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f\": container with ID starting with f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f not found: ID does not exist" containerID="f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f" Sep 10 19:51:05.719746 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.719707 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f"} err="failed to get container status \"f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f\": rpc error: code = NotFound desc = could not find container \"f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f\": container with ID starting with f35b726ed25431800e544de6be92a07185bf2519356040172d829c7d28b6779f not found: ID does not exist" Sep 10 19:51:05.719746 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.719726 2584 scope.go:117] "RemoveContainer" containerID="f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34" Sep 10 19:51:05.720009 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:51:05.719991 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34\": container with ID starting with f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34 not found: ID does not exist" containerID="f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34" Sep 10 19:51:05.720052 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.720015 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34"} err="failed to get container status \"f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34\": rpc error: code = NotFound desc = could not find container \"f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34\": container with ID starting with f7a39d4110f53f2e400816c0090122ff09ae5a630e8bece50899ead7097ffd34 not found: ID does not exist" Sep 10 19:51:05.723914 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.723885 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp"] Sep 10 19:51:05.727878 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:05.727850 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-07c5fe-openshift-default-5f9ccb65dd-kxwgp"] Sep 10 19:51:06.488667 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:06.488630 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" path="/var/lib/kubelet/pods/361212c6-d3d6-454f-ae24-f2882b1e3469/volumes" Sep 10 19:51:10.689293 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:10.689239 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q"] Sep 10 19:51:10.690958 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:10.690927 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1fa9-29817825-wgl5q"] Sep 10 19:51:10.722855 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:10.722822 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4"] Sep 10 19:51:10.723139 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:10.723087 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" podUID="f87f0afe-5140-4636-a690-bbc13cf75d78" containerName="payload-pre-processing" containerID="cri-o://2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8" gracePeriod=30 Sep 10 19:51:10.973545 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:10.973516 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:51:11.023215 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.023179 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87f0afe-5140-4636-a690-bbc13cf75d78-plugins-config-volume\") pod \"f87f0afe-5140-4636-a690-bbc13cf75d78\" (UID: \"f87f0afe-5140-4636-a690-bbc13cf75d78\") " Sep 10 19:51:11.023442 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.023250 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65rxg\" (UniqueName: \"kubernetes.io/projected/f87f0afe-5140-4636-a690-bbc13cf75d78-kube-api-access-65rxg\") pod \"f87f0afe-5140-4636-a690-bbc13cf75d78\" (UID: \"f87f0afe-5140-4636-a690-bbc13cf75d78\") " Sep 10 19:51:11.023665 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.023640 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87f0afe-5140-4636-a690-bbc13cf75d78-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f87f0afe-5140-4636-a690-bbc13cf75d78" (UID: "f87f0afe-5140-4636-a690-bbc13cf75d78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:51:11.025435 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.025404 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87f0afe-5140-4636-a690-bbc13cf75d78-kube-api-access-65rxg" (OuterVolumeSpecName: "kube-api-access-65rxg") pod "f87f0afe-5140-4636-a690-bbc13cf75d78" (UID: "f87f0afe-5140-4636-a690-bbc13cf75d78"). InnerVolumeSpecName "kube-api-access-65rxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:51:11.124127 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.124092 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65rxg\" (UniqueName: \"kubernetes.io/projected/f87f0afe-5140-4636-a690-bbc13cf75d78-kube-api-access-65rxg\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:11.124127 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.124121 2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87f0afe-5140-4636-a690-bbc13cf75d78-plugins-config-volume\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:11.734219 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.734182 2584 generic.go:356] "Generic (PLEG): container finished" podID="f87f0afe-5140-4636-a690-bbc13cf75d78" containerID="2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8" exitCode=0 Sep 10 19:51:11.734639 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.734227 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" Sep 10 19:51:11.734639 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.734269 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" event={"ID":"f87f0afe-5140-4636-a690-bbc13cf75d78","Type":"ContainerDied","Data":"2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8"} Sep 10 19:51:11.734639 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.734349 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" event={"ID":"f87f0afe-5140-4636-a690-bbc13cf75d78","Type":"ContainerDied","Data":"6810a903c03f558bf8c3ea20bfc39874a2c7188202337a7d15f1f57ebfc8174b"} Sep 10 19:51:11.734639 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.734378 2584 scope.go:117] "RemoveContainer" containerID="2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8" Sep 10 19:51:11.743577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.743554 2584 scope.go:117] "RemoveContainer" containerID="2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8" Sep 10 19:51:11.743859 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:51:11.743838 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8\": container with ID starting with 2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8 not found: ID does not exist" containerID="2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8" Sep 10 19:51:11.743903 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.743867 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8"} err="failed to get container status \"2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8\": rpc error: code = NotFound desc = could not find container \"2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8\": container with ID starting with 2ea6a3c18ea78e247769bba188777844234feadd5690abb6b5f862bfaa6cbeb8 not found: ID does not exist" Sep 10 19:51:11.757126 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.757092 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4"] Sep 10 19:51:11.760672 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.760643 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4"] Sep 10 19:51:11.910599 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:11.910552 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1fa9-5dd5d88b86-fqpr4" podUID="f87f0afe-5140-4636-a690-bbc13cf75d78" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.50:9004: i/o timeout" Sep 10 19:51:12.488273 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:12.488236 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3d36e27-57a1-492e-9cc4-8da9c0f78b8e" path="/var/lib/kubelet/pods/c3d36e27-57a1-492e-9cc4-8da9c0f78b8e/volumes" Sep 10 19:51:12.488655 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:12.488640 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f87f0afe-5140-4636-a690-bbc13cf75d78" path="/var/lib/kubelet/pods/f87f0afe-5140-4636-a690-bbc13cf75d78/volumes" Sep 10 19:51:14.750114 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:14.750071 2584 scope.go:117] "RemoveContainer" containerID="cfea5e6964fe7ec76123e3cf28e50c0828e889d061167bfda16c1ac3a0079806" Sep 10 19:51:14.758567 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:14.758544 2584 scope.go:117] "RemoveContainer" containerID="054ea0e2689f3adbdc7508d79f92ece8f46f64281965a968316701bc40f9c9cf" Sep 10 19:51:17.861610 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:17.861578 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh"] Sep 10 19:51:17.862121 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:17.861800 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" containerID="cri-o://cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7" gracePeriod=30 Sep 10 19:51:23.114354 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.114321 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:51:23.240529 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240423 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfac205e-0cb1-426c-8411-1dad1d224947-istiod-ca-cert\") pod \"cfac205e-0cb1-426c-8411-1dad1d224947\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " Sep 10 19:51:23.240529 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240473 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-envoy\") pod \"cfac205e-0cb1-426c-8411-1dad1d224947\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " Sep 10 19:51:23.240529 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240493 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfac205e-0cb1-426c-8411-1dad1d224947-istio-podinfo\") pod \"cfac205e-0cb1-426c-8411-1dad1d224947\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " Sep 10 19:51:23.240529 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240532 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmjvd\" (UniqueName: \"kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-kube-api-access-qmjvd\") pod \"cfac205e-0cb1-426c-8411-1dad1d224947\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " Sep 10 19:51:23.240853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240568 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-certs\") pod \"cfac205e-0cb1-426c-8411-1dad1d224947\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " Sep 10 19:51:23.240853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240625 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-credential-socket\") pod \"cfac205e-0cb1-426c-8411-1dad1d224947\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " Sep 10 19:51:23.240853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240658 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-istio-token\") pod \"cfac205e-0cb1-426c-8411-1dad1d224947\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " Sep 10 19:51:23.240853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240719 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-socket\") pod \"cfac205e-0cb1-426c-8411-1dad1d224947\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " Sep 10 19:51:23.240853 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240748 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-data\") pod \"cfac205e-0cb1-426c-8411-1dad1d224947\" (UID: \"cfac205e-0cb1-426c-8411-1dad1d224947\") " Sep 10 19:51:23.241132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240917 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfac205e-0cb1-426c-8411-1dad1d224947-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cfac205e-0cb1-426c-8411-1dad1d224947" (UID: "cfac205e-0cb1-426c-8411-1dad1d224947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:51:23.241132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240966 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cfac205e-0cb1-426c-8411-1dad1d224947" (UID: "cfac205e-0cb1-426c-8411-1dad1d224947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:51:23.241132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.240990 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cfac205e-0cb1-426c-8411-1dad1d224947" (UID: "cfac205e-0cb1-426c-8411-1dad1d224947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:51:23.241132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.241105 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cfac205e-0cb1-426c-8411-1dad1d224947" (UID: "cfac205e-0cb1-426c-8411-1dad1d224947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:51:23.241395 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.241138 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:23.241395 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.241142 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-data" (OuterVolumeSpecName: "istio-data") pod "cfac205e-0cb1-426c-8411-1dad1d224947" (UID: "cfac205e-0cb1-426c-8411-1dad1d224947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:51:23.241395 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.241157 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-credential-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:23.241395 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.241172 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfac205e-0cb1-426c-8411-1dad1d224947-istiod-ca-cert\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:23.242804 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.242767 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cfac205e-0cb1-426c-8411-1dad1d224947" (UID: "cfac205e-0cb1-426c-8411-1dad1d224947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:51:23.242918 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.242859 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfac205e-0cb1-426c-8411-1dad1d224947-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cfac205e-0cb1-426c-8411-1dad1d224947" (UID: "cfac205e-0cb1-426c-8411-1dad1d224947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:51:23.242963 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.242910 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-istio-token" (OuterVolumeSpecName: "istio-token") pod "cfac205e-0cb1-426c-8411-1dad1d224947" (UID: "cfac205e-0cb1-426c-8411-1dad1d224947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:51:23.243012 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.242995 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-kube-api-access-qmjvd" (OuterVolumeSpecName: "kube-api-access-qmjvd") pod "cfac205e-0cb1-426c-8411-1dad1d224947" (UID: "cfac205e-0cb1-426c-8411-1dad1d224947"). InnerVolumeSpecName "kube-api-access-qmjvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:51:23.341756 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.341713 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-istio-token\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:23.341756 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.341744 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-workload-socket\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:23.341756 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.341753 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-data\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:23.341756 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.341762 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfac205e-0cb1-426c-8411-1dad1d224947-istio-envoy\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:23.341756 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.341769 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfac205e-0cb1-426c-8411-1dad1d224947-istio-podinfo\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:23.342064 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.341778 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmjvd\" (UniqueName: \"kubernetes.io/projected/cfac205e-0cb1-426c-8411-1dad1d224947-kube-api-access-qmjvd\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:51:23.800115 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.800079 2584 generic.go:356] "Generic (PLEG): container finished" podID="cfac205e-0cb1-426c-8411-1dad1d224947" containerID="cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7" exitCode=0 Sep 10 19:51:23.800345 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.800127 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" Sep 10 19:51:23.800345 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.800167 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" event={"ID":"cfac205e-0cb1-426c-8411-1dad1d224947","Type":"ContainerDied","Data":"cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7"} Sep 10 19:51:23.800345 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.800235 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh" event={"ID":"cfac205e-0cb1-426c-8411-1dad1d224947","Type":"ContainerDied","Data":"357981a52602d358b568946f6c8077fc00ce85ac7c44632ac3afd41698127d20"} Sep 10 19:51:23.800345 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.800270 2584 scope.go:117] "RemoveContainer" containerID="cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7" Sep 10 19:51:23.809360 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.809336 2584 scope.go:117] "RemoveContainer" containerID="f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7" Sep 10 19:51:23.818062 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.818043 2584 scope.go:117] "RemoveContainer" containerID="cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7" Sep 10 19:51:23.818408 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:51:23.818388 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7\": container with ID starting with cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7 not found: ID does not exist" containerID="cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7" Sep 10 19:51:23.818464 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.818418 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7"} err="failed to get container status \"cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7\": rpc error: code = NotFound desc = could not find container \"cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7\": container with ID starting with cf71b1c055c510bbcd35be5895f99254dc3c1ec1be3eaa7acf93d77c144ff6d7 not found: ID does not exist" Sep 10 19:51:23.818464 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.818435 2584 scope.go:117] "RemoveContainer" containerID="f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7" Sep 10 19:51:23.818692 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:51:23.818674 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7\": container with ID starting with f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7 not found: ID does not exist" containerID="f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7" Sep 10 19:51:23.818749 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.818703 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7"} err="failed to get container status \"f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7\": rpc error: code = NotFound desc = could not find container \"f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7\": container with ID starting with f6a317994409499cb357c34381d6ef08f3dfe733396a806b4378abadfbd261f7 not found: ID does not exist" Sep 10 19:51:23.821808 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.821784 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh"] Sep 10 19:51:23.825484 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:23.825458 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1fa9-openshift-default-55b95d4d7d-65fqh"] Sep 10 19:51:24.491205 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:51:24.491157 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" path="/var/lib/kubelet/pods/cfac205e-0cb1-426c-8411-1dad1d224947/volumes" Sep 10 19:55:14.824157 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:14.824122 2584 scope.go:117] "RemoveContainer" containerID="aee47c3429d43449a57b86246dd476244d7d6c7d1c2d1864a9f82aeab2d83fda" Sep 10 19:55:37.800409 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800368 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n"] Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800719 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87f0afe-5140-4636-a690-bbc13cf75d78" containerName="payload-pre-processing" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800732 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87f0afe-5140-4636-a690-bbc13cf75d78" containerName="payload-pre-processing" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800744 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7577377e-99bc-4b49-9de6-0596d8b4a367" containerName="revoke-keys" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800751 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7577377e-99bc-4b49-9de6-0596d8b4a367" containerName="revoke-keys" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800767 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800774 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800786 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800791 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800798 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800802 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800808 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800813 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800861 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800870 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800877 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfac205e-0cb1-426c-8411-1dad1d224947" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800882 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="361212c6-d3d6-454f-ae24-f2882b1e3469" containerName="istio-proxy" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800888 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7577377e-99bc-4b49-9de6-0596d8b4a367" containerName="revoke-keys" Sep 10 19:55:37.800900 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.800895 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87f0afe-5140-4636-a690-bbc13cf75d78" containerName="payload-pre-processing" Sep 10 19:55:37.803737 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.803719 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.806380 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.806349 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fd3007ef-kserve-self-signed-certs\"" Sep 10 19:55:37.816424 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.816396 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n"] Sep 10 19:55:37.896128 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.896085 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-model-cache\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.896128 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.896130 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-dshm\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.896442 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.896153 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tmp-dir\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.896442 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.896238 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tls-certs\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.896442 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.896376 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-home\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.896442 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.896416 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znb4n\" (UniqueName: \"kubernetes.io/projected/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-kube-api-access-znb4n\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.997928 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.997877 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-dshm\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.997928 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.997928 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tmp-dir\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.998192 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.997967 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tls-certs\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.998192 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.998041 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-home\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.998192 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.998077 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znb4n\" (UniqueName: \"kubernetes.io/projected/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-kube-api-access-znb4n\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.998192 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.998131 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-model-cache\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.998468 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.998443 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tmp-dir\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.998539 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.998481 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-home\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:37.998591 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:37.998540 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-model-cache\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:38.000287 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:38.000251 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-dshm\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:38.000547 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:38.000531 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tls-certs\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:38.007991 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:38.007966 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znb4n\" (UniqueName: \"kubernetes.io/projected/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-kube-api-access-znb4n\") pod \"e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:38.113177 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:38.113135 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:38.394093 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:38.394063 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n"] Sep 10 19:55:38.395963 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:55:38.395933 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod850edbc8_5771_4d0b_ba8a_c93a0880ceaa.slice/crio-5a0569a5652846eebcccab42957c08adfeb6aeb25037b3176b296a9edec217f9 WatchSource:0}: Error finding container 5a0569a5652846eebcccab42957c08adfeb6aeb25037b3176b296a9edec217f9: Status 404 returned error can't find the container with id 5a0569a5652846eebcccab42957c08adfeb6aeb25037b3176b296a9edec217f9 Sep 10 19:55:39.033485 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:39.033440 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" event={"ID":"850edbc8-5771-4d0b-ba8a-c93a0880ceaa","Type":"ContainerStarted","Data":"5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa"} Sep 10 19:55:39.033887 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:39.033496 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" event={"ID":"850edbc8-5771-4d0b-ba8a-c93a0880ceaa","Type":"ContainerStarted","Data":"5a0569a5652846eebcccab42957c08adfeb6aeb25037b3176b296a9edec217f9"} Sep 10 19:55:39.052355 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:39.052290 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" podStartSLOduration=2.052257547 podStartE2EDuration="2.052257547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:39.050244222 +0000 UTC m=+2125.018056470" watchObservedRunningTime="2026-09-10 19:55:39.052257547 +0000 UTC m=+2125.020069794" Sep 10 19:55:48.113778 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:48.113732 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:48.114263 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:48.113794 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:48.126504 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:48.126472 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:55:49.093385 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:55:49.093352 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:56:06.452791 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:06.452753 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n"] Sep 10 19:56:21.472398 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.472307 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" podUID="850edbc8-5771-4d0b-ba8a-c93a0880ceaa" containerName="main" containerID="cri-o://5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa" gracePeriod=45 Sep 10 19:56:21.724518 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.724449 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:56:21.801917 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.801872 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-dshm\") pod \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " Sep 10 19:56:21.802139 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.801950 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tls-certs\") pod \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " Sep 10 19:56:21.802139 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.801989 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tmp-dir\") pod \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " Sep 10 19:56:21.802139 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.802012 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-home\") pod \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " Sep 10 19:56:21.802139 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.802045 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znb4n\" (UniqueName: \"kubernetes.io/projected/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-kube-api-access-znb4n\") pod \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " Sep 10 19:56:21.802139 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.802081 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-model-cache\") pod \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\" (UID: \"850edbc8-5771-4d0b-ba8a-c93a0880ceaa\") " Sep 10 19:56:21.802438 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.802323 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-home" (OuterVolumeSpecName: "home") pod "850edbc8-5771-4d0b-ba8a-c93a0880ceaa" (UID: "850edbc8-5771-4d0b-ba8a-c93a0880ceaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:56:21.802438 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.802331 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "850edbc8-5771-4d0b-ba8a-c93a0880ceaa" (UID: "850edbc8-5771-4d0b-ba8a-c93a0880ceaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:56:21.802546 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.802462 2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tmp-dir\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:56:21.802546 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.802475 2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-home\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:56:21.802546 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.802468 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-model-cache" (OuterVolumeSpecName: "model-cache") pod "850edbc8-5771-4d0b-ba8a-c93a0880ceaa" (UID: "850edbc8-5771-4d0b-ba8a-c93a0880ceaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:56:21.804265 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.804233 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-kube-api-access-znb4n" (OuterVolumeSpecName: "kube-api-access-znb4n") pod "850edbc8-5771-4d0b-ba8a-c93a0880ceaa" (UID: "850edbc8-5771-4d0b-ba8a-c93a0880ceaa"). InnerVolumeSpecName "kube-api-access-znb4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:56:21.804437 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.804305 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-dshm" (OuterVolumeSpecName: "dshm") pod "850edbc8-5771-4d0b-ba8a-c93a0880ceaa" (UID: "850edbc8-5771-4d0b-ba8a-c93a0880ceaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:56:21.804437 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.804333 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "850edbc8-5771-4d0b-ba8a-c93a0880ceaa" (UID: "850edbc8-5771-4d0b-ba8a-c93a0880ceaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:56:21.902930 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.902890 2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-tls-certs\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:56:21.902930 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.902922 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znb4n\" (UniqueName: \"kubernetes.io/projected/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-kube-api-access-znb4n\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:56:21.902930 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.902934 2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-model-cache\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:56:21.903174 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:21.902943 2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850edbc8-5771-4d0b-ba8a-c93a0880ceaa-dshm\") on node \"ip-10-0-134-64.ec2.internal\" DevicePath \"\"" Sep 10 19:56:22.242356 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:22.242314 2584 generic.go:356] "Generic (PLEG): container finished" podID="850edbc8-5771-4d0b-ba8a-c93a0880ceaa" containerID="5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa" exitCode=0 Sep 10 19:56:22.242356 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:22.242360 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" Sep 10 19:56:22.242636 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:22.242396 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" event={"ID":"850edbc8-5771-4d0b-ba8a-c93a0880ceaa","Type":"ContainerDied","Data":"5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa"} Sep 10 19:56:22.242636 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:22.242448 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n" event={"ID":"850edbc8-5771-4d0b-ba8a-c93a0880ceaa","Type":"ContainerDied","Data":"5a0569a5652846eebcccab42957c08adfeb6aeb25037b3176b296a9edec217f9"} Sep 10 19:56:22.242636 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:22.242472 2584 scope.go:117] "RemoveContainer" containerID="5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa" Sep 10 19:56:22.254948 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:22.254912 2584 scope.go:117] "RemoveContainer" containerID="5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa" Sep 10 19:56:22.255403 ip-10-0-134-64 kubenswrapper[2584]: E0910 19:56:22.255371 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa\": container with ID starting with 5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa not found: ID does not exist" containerID="5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa" Sep 10 19:56:22.255610 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:22.255561 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa"} err="failed to get container status \"5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa\": rpc error: code = NotFound desc = could not find container \"5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa\": container with ID starting with 5a665e9fdbf07e1d7eed316daf56dec4562796bada4f4f13a7216b7b13e0f5aa not found: ID does not exist" Sep 10 19:56:22.269364 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:22.269324 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n"] Sep 10 19:56:22.273168 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:22.273136 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fd3007ef-kserve-565fbb4b8c-5x69n"] Sep 10 19:56:22.489864 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:22.489821 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="850edbc8-5771-4d0b-ba8a-c93a0880ceaa" path="/var/lib/kubelet/pods/850edbc8-5771-4d0b-ba8a-c93a0880ceaa/volumes" Sep 10 19:56:39.873006 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.872968 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj"] Sep 10 19:56:39.873410 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.873335 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850edbc8-5771-4d0b-ba8a-c93a0880ceaa" containerName="main" Sep 10 19:56:39.873410 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.873351 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="850edbc8-5771-4d0b-ba8a-c93a0880ceaa" containerName="main" Sep 10 19:56:39.873486 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.873434 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="850edbc8-5771-4d0b-ba8a-c93a0880ceaa" containerName="main" Sep 10 19:56:39.878250 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.878219 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:39.881670 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.881643 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 19:56:39.881876 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.881692 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 19:56:39.881876 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.881643 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:56:39.881876 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.881643 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 19:56:39.881876 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.881643 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5cfmg\"" Sep 10 19:56:39.887788 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.887757 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj"] Sep 10 19:56:39.965236 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.965190 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690a2980-26b6-4a36-8054-57461876f9d8-metrics-certs\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:39.965438 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.965289 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5cdb\" (UniqueName: \"kubernetes.io/projected/690a2980-26b6-4a36-8054-57461876f9d8-kube-api-access-v5cdb\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:39.965438 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.965375 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690a2980-26b6-4a36-8054-57461876f9d8-openshift-service-ca\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:39.965438 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:39.965421 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690a2980-26b6-4a36-8054-57461876f9d8-maas-api-tls\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:40.066346 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.066306 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690a2980-26b6-4a36-8054-57461876f9d8-openshift-service-ca\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:40.066524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.066358 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690a2980-26b6-4a36-8054-57461876f9d8-maas-api-tls\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:40.066524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.066420 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690a2980-26b6-4a36-8054-57461876f9d8-metrics-certs\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:40.066524 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.066440 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5cdb\" (UniqueName: \"kubernetes.io/projected/690a2980-26b6-4a36-8054-57461876f9d8-kube-api-access-v5cdb\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:40.067061 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.067033 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690a2980-26b6-4a36-8054-57461876f9d8-openshift-service-ca\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:40.068968 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.068945 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690a2980-26b6-4a36-8054-57461876f9d8-maas-api-tls\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:40.069060 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.068989 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690a2980-26b6-4a36-8054-57461876f9d8-metrics-certs\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:40.075327 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.075299 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5cdb\" (UniqueName: \"kubernetes.io/projected/690a2980-26b6-4a36-8054-57461876f9d8-kube-api-access-v5cdb\") pod \"maas-api-698bfd7f85-96zhj\" (UID: \"690a2980-26b6-4a36-8054-57461876f9d8\") " pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:40.190568 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.190478 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:40.471541 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.471511 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj"] Sep 10 19:56:40.474604 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:56:40.474576 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690a2980_26b6_4a36_8054_57461876f9d8.slice/crio-e6f98933aea3f358a806936b7dc4a78d7f99a58c450139a353d5001ce2c33e11 WatchSource:0}: Error finding container e6f98933aea3f358a806936b7dc4a78d7f99a58c450139a353d5001ce2c33e11: Status 404 returned error can't find the container with id e6f98933aea3f358a806936b7dc4a78d7f99a58c450139a353d5001ce2c33e11 Sep 10 19:56:40.476753 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:40.476730 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:56:41.343926 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:41.343877 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" event={"ID":"690a2980-26b6-4a36-8054-57461876f9d8","Type":"ContainerStarted","Data":"e6f98933aea3f358a806936b7dc4a78d7f99a58c450139a353d5001ce2c33e11"} Sep 10 19:56:43.357157 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:43.357118 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" event={"ID":"690a2980-26b6-4a36-8054-57461876f9d8","Type":"ContainerStarted","Data":"f91fda4da2036af08dc4a5d38c7180c54b5c530560bc5e7892614ae1d75c007a"} Sep 10 19:56:43.357587 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:43.357309 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:56:43.375785 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:43.375723 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" podStartSLOduration=2.084834637 podStartE2EDuration="4.375704506s" totalImagesPullingTime="2.290869869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:56:43.373922792 +0000 UTC m=+2189.341735039" watchObservedRunningTime="2026-09-10 19:56:43.375704506 +0000 UTC m=+2189.343516753" Sep 10 19:56:49.368054 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:56:49.368018 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-698bfd7f85-96zhj" Sep 10 19:57:07.800503 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:07.800464 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xx9l8_3d18976d-a5b7-484c-b36d-6bde0096be99/manager/0.log" Sep 10 19:57:07.927679 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:07.927643 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q556f_07c67934-a96f-45de-b287-a4933090904d/manager/0.log" Sep 10 19:57:08.200920 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:08.200888 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-md5v2_fbd116bd-2928-48ef-9990-b3766580e6c7/server/0.log" Sep 10 19:57:08.716525 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:08.716493 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ddcddd787-wmrjz_19a7ac54-3c02-42ac-959a-a37752df52f7/copy-manifests/0.log" Sep 10 19:57:08.723660 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:08.723634 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ddcddd787-wmrjz_19a7ac54-3c02-42ac-959a-a37752df52f7/manager/0.log" Sep 10 19:57:09.183159 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:09.183125 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-698bfd7f85-96zhj_690a2980-26b6-4a36-8054-57461876f9d8/maas-api/0.log" Sep 10 19:57:09.301248 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:09.301216 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817825-nw9ms_04d9faa3-0e8f-4225-987d-4548a8970ebd/cleanup/0.log" Sep 10 19:57:11.017312 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:11.017268 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-zkph9_b81f8072-fd40-4342-85cb-afe7956ddb62/manager/0.log" Sep 10 19:57:11.403729 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:11.403697 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj_4f37593c-597c-41ac-ae4c-4f4e3a819ba5/istio-proxy/0.log" Sep 10 19:57:11.882815 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:11.882781 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7646x_ac470cae-838a-4175-bd45-be2ba7424eca/istio-proxy/1.log" Sep 10 19:57:12.011408 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:12.011373 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-krgbg_8c49cb9e-851e-4b1e-9ad1-3acb1c67038c/payload-pre-processing/0.log" Sep 10 19:57:12.764331 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:12.764273 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-dxnld_8433d3fe-d47b-46c1-8811-28dc17a1325b/main/0.log" Sep 10 19:57:13.113492 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:13.113463 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk9n5b_ad91ab45-5705-4499-9a03-e3ca53cdb107/main/0.log" Sep 10 19:57:13.368755 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:13.368660 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xphq4_f890f0f7-cdbc-4ff3-870b-b9479ac0261e/main/0.log" Sep 10 19:57:14.849743 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:14.849712 2584 scope.go:117] "RemoveContainer" containerID="ad1de8c5602e1704f4d8e57c353b7621f37bc957b52e152195c3c1bf364110b1" Sep 10 19:57:14.858028 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:14.858003 2584 scope.go:117] "RemoveContainer" containerID="2751a6d36730c46dabe88c25a4dc44caf1f2a70ac5f5e21d2ea503039731d702" Sep 10 19:57:22.127571 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:22.127531 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jclqb_9dae04ef-d3ba-4c10-8fc6-9eef8093c500/global-pull-secret-syncer/0.log" Sep 10 19:57:22.300709 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:22.300667 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f89b2_d45fd229-64fe-4f71-a53f-d20d4591c8df/konnectivity-agent/0.log" Sep 10 19:57:22.377155 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:22.377123 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-64.ec2.internal_a45295e4379262bb71d3ff91072070b1/haproxy/0.log" Sep 10 19:57:27.690989 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:27.690951 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-zkph9_b81f8072-fd40-4342-85cb-afe7956ddb62/manager/0.log" Sep 10 19:57:30.008117 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:30.008025 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kh7qh_615528aa-2d5f-47fe-80a4-5d83fbeac9a3/node-exporter/0.log" Sep 10 19:57:30.108216 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:30.108181 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kh7qh_615528aa-2d5f-47fe-80a4-5d83fbeac9a3/kube-rbac-proxy/0.log" Sep 10 19:57:30.188071 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:30.188044 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kh7qh_615528aa-2d5f-47fe-80a4-5d83fbeac9a3/init-textfile/0.log" Sep 10 19:57:31.155577 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:31.155542 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687966b585-xd8ng_ab0cc04f-dee0-45e2-8e78-ed57c3d60cce/thanos-query/0.log" Sep 10 19:57:31.182009 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:31.181980 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687966b585-xd8ng_ab0cc04f-dee0-45e2-8e78-ed57c3d60cce/kube-rbac-proxy-web/0.log" Sep 10 19:57:31.215158 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:31.215121 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687966b585-xd8ng_ab0cc04f-dee0-45e2-8e78-ed57c3d60cce/kube-rbac-proxy/0.log" Sep 10 19:57:31.257177 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:31.257141 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687966b585-xd8ng_ab0cc04f-dee0-45e2-8e78-ed57c3d60cce/prom-label-proxy/0.log" Sep 10 19:57:31.281651 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:31.281615 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687966b585-xd8ng_ab0cc04f-dee0-45e2-8e78-ed57c3d60cce/kube-rbac-proxy-rules/0.log" Sep 10 19:57:31.318482 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:31.318448 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687966b585-xd8ng_ab0cc04f-dee0-45e2-8e78-ed57c3d60cce/kube-rbac-proxy-metrics/0.log" Sep 10 19:57:32.405450 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.405413 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv"] Sep 10 19:57:32.408330 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.408303 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.411737 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.411711 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lkczj\"/\"kube-root-ca.crt\"" Sep 10 19:57:32.412160 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.412140 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lkczj\"/\"default-dockercfg-rr7jr\"" Sep 10 19:57:32.413328 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.413309 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lkczj\"/\"openshift-service-ca.crt\"" Sep 10 19:57:32.425381 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.425352 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv"] Sep 10 19:57:32.538643 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.538604 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-lib-modules\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.538643 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.538642 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-sys\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.538881 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.538681 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-proc\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.538881 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.538699 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c6vv\" (UniqueName: \"kubernetes.io/projected/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-kube-api-access-9c6vv\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.538881 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.538784 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-podres\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.639482 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.639438 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-proc\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.639482 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.639480 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c6vv\" (UniqueName: \"kubernetes.io/projected/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-kube-api-access-9c6vv\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.639759 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.639543 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-podres\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.639759 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.639565 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-lib-modules\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.639759 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.639567 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-proc\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.639759 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.639615 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-sys\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.639759 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.639667 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-podres\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.639759 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.639722 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-sys\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.640040 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.639768 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-lib-modules\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.649063 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.649033 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c6vv\" (UniqueName: \"kubernetes.io/projected/cb7fdfc9-7f26-47a6-a55c-7f443893ab1e-kube-api-access-9c6vv\") pod \"perf-node-gather-daemonset-h2vvv\" (UID: \"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e\") " pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.718766 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.718679 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:32.992930 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:32.992843 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv"] Sep 10 19:57:32.996627 ip-10-0-134-64 kubenswrapper[2584]: W0910 19:57:32.996595 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb7fdfc9_7f26_47a6_a55c_7f443893ab1e.slice/crio-75642254dfa2725b31324e7215158faeedf1d657511aaa5025586ed9b106e66b WatchSource:0}: Error finding container 75642254dfa2725b31324e7215158faeedf1d657511aaa5025586ed9b106e66b: Status 404 returned error can't find the container with id 75642254dfa2725b31324e7215158faeedf1d657511aaa5025586ed9b106e66b Sep 10 19:57:33.594024 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:33.593979 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" event={"ID":"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e","Type":"ContainerStarted","Data":"19cce3e89aab31cccb46203000399c3cae1380236c28226a12eeff56e076d9d7"} Sep 10 19:57:33.594466 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:33.594135 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:33.594466 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:33.594175 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" event={"ID":"cb7fdfc9-7f26-47a6-a55c-7f443893ab1e","Type":"ContainerStarted","Data":"75642254dfa2725b31324e7215158faeedf1d657511aaa5025586ed9b106e66b"} Sep 10 19:57:33.612035 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:33.611974 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" podStartSLOduration=1.6119552929999998 podStartE2EDuration="1.611955293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:57:33.611123116 +0000 UTC m=+2239.578935373" watchObservedRunningTime="2026-09-10 19:57:33.611955293 +0000 UTC m=+2239.579767540" Sep 10 19:57:34.060059 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:34.060027 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-645c954fbc-gscd9_111b8e3c-01f1-4797-9164-544f8905a7f8/console/0.log" Sep 10 19:57:35.988496 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:35.988461 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-klkw2_974975b6-d004-4708-a3d3-1757891d57a0/dns/0.log" Sep 10 19:57:36.015423 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:36.015390 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-klkw2_974975b6-d004-4708-a3d3-1757891d57a0/kube-rbac-proxy/0.log" Sep 10 19:57:36.223545 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:36.223516 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jfqm6_1a11e036-f7b7-4856-ad1a-96a0872133e5/dns-node-resolver/0.log" Sep 10 19:57:36.934195 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:36.934115 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jzlhg_2ed20544-9b54-4e94-9ebb-7cf426aabff9/node-ca/0.log" Sep 10 19:57:38.058690 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:38.058658 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj_4f37593c-597c-41ac-ae4c-4f4e3a819ba5/istio-proxy/0.log" Sep 10 19:57:38.368159 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:38.368075 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7646x_ac470cae-838a-4175-bd45-be2ba7424eca/istio-proxy/0.log" Sep 10 19:57:38.375877 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:38.375847 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7646x_ac470cae-838a-4175-bd45-be2ba7424eca/istio-proxy/1.log" Sep 10 19:57:38.410415 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:38.410375 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-krgbg_8c49cb9e-851e-4b1e-9ad1-3acb1c67038c/payload-pre-processing/0.log" Sep 10 19:57:39.436172 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:39.436140 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wv5m9_23f79489-278b-41b0-b002-474e6da0dc7c/serve-healthcheck-canary/0.log" Sep 10 19:57:39.609132 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:39.609089 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lkczj/perf-node-gather-daemonset-h2vvv" Sep 10 19:57:40.070619 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:40.070585 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzjrz_56353bf4-bd0a-4710-9f0e-35d21ddda90b/kube-rbac-proxy/0.log" Sep 10 19:57:40.095446 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:40.095413 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzjrz_56353bf4-bd0a-4710-9f0e-35d21ddda90b/exporter/0.log" Sep 10 19:57:40.120335 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:40.120305 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzjrz_56353bf4-bd0a-4710-9f0e-35d21ddda90b/extractor/0.log" Sep 10 19:57:42.605629 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:42.605596 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xx9l8_3d18976d-a5b7-484c-b36d-6bde0096be99/manager/0.log" Sep 10 19:57:42.800941 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:42.800848 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q556f_07c67934-a96f-45de-b287-a4933090904d/manager/0.log" Sep 10 19:57:42.961355 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:42.961326 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-md5v2_fbd116bd-2928-48ef-9990-b3766580e6c7/server/0.log" Sep 10 19:57:43.248988 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:43.248959 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ddcddd787-wmrjz_19a7ac54-3c02-42ac-959a-a37752df52f7/manager/0.log" Sep 10 19:57:43.274026 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:43.273996 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ddcddd787-wmrjz_19a7ac54-3c02-42ac-959a-a37752df52f7/copy-manifests/0.log" Sep 10 19:57:43.939337 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:43.939309 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj_4f37593c-597c-41ac-ae4c-4f4e3a819ba5/istio-proxy/0.log" Sep 10 19:57:44.217245 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:44.217147 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7646x_ac470cae-838a-4175-bd45-be2ba7424eca/istio-proxy/0.log" Sep 10 19:57:44.225522 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:44.225492 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7646x_ac470cae-838a-4175-bd45-be2ba7424eca/istio-proxy/1.log" Sep 10 19:57:44.247176 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:44.247143 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-krgbg_8c49cb9e-851e-4b1e-9ad1-3acb1c67038c/payload-pre-processing/0.log" Sep 10 19:57:44.665987 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:44.665957 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57fccbbc8-k9nvb_a491f76f-cfef-499a-9b1c-b11615af46f0/manager/0.log" Sep 10 19:57:44.694141 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:44.694108 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-s4qrz_5f6a3d7f-f703-49d5-a1d9-b8d6c0ceac13/openshift-lws-operator/0.log" Sep 10 19:57:45.862836 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:45.862800 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xhltj_4f37593c-597c-41ac-ae4c-4f4e3a819ba5/istio-proxy/0.log" Sep 10 19:57:46.140525 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:46.140423 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7646x_ac470cae-838a-4175-bd45-be2ba7424eca/istio-proxy/0.log" Sep 10 19:57:46.148258 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:46.148215 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7646x_ac470cae-838a-4175-bd45-be2ba7424eca/istio-proxy/1.log" Sep 10 19:57:46.173955 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:46.173903 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-krgbg_8c49cb9e-851e-4b1e-9ad1-3acb1c67038c/payload-pre-processing/0.log" Sep 10 19:57:50.151720 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:50.151690 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-d845v_9eab30f3-754a-47c2-a5e4-1c1e0faab235/migrator/0.log" Sep 10 19:57:50.176907 ip-10-0-134-64 kubenswrapper[2584]: I0910 19:57:50.176874 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-d845v_9eab30f3-754a-47c2-a5e4-1c1e0faab235/graceful-termination/0.log"