Sep 15 20:34:46.287582 ip-10-0-130-107 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 15 20:34:46.287599 ip-10-0-130-107 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 15 20:34:46.287610 ip-10-0-130-107 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 15 20:34:46.287957 ip-10-0-130-107 systemd[1]: Failed to start Kubernetes Kubelet. Sep 15 20:34:56.308026 ip-10-0-130-107 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 15 20:34:56.308040 ip-10-0-130-107 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ecbaf48cebf641f2b6e2da096ff2982f -- Sep 15 20:36:42.674881 ip-10-0-130-107 systemd[1]: Starting Kubernetes Kubelet... Sep 15 20:36:43.127534 ip-10-0-130-107 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 20:36:43.127534 ip-10-0-130-107 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 15 20:36:43.127534 ip-10-0-130-107 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 20:36:43.127534 ip-10-0-130-107 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 15 20:36:43.127534 ip-10-0-130-107 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 20:36:43.128292 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.127598 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 15 20:36:43.133639 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133607 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:43.133639 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133630 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:43.133639 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133634 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:43.133639 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133637 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:43.133639 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133640 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:43.133639 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133644 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:43.133639 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133648 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133651 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133653 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133656 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133659 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133662 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133665 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133667 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133672 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133676 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133679 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133682 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133685 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133688 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133691 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133694 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133696 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133699 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133701 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:43.133889 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133704 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133707 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133709 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133713 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133715 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133718 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133720 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133723 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133725 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133729 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133732 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133734 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133737 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133740 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133742 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133746 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133749 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133752 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133754 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133757 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:43.134382 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133759 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133762 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133765 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133767 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133770 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133772 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133775 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133777 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133780 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133782 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133785 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133787 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133790 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133792 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133795 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133797 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133800 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133803 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133805 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133808 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:43.134915 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133810 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133813 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133816 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133818 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133821 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133823 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133827 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133832 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133835 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133837 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133840 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133843 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133845 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133849 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133852 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133855 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133857 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133860 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133863 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:43.135402 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133865 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133869 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133873 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133876 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133879 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133881 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133884 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133887 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133890 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133892 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133895 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133898 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133900 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133903 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.133906 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135474 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135485 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135488 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135491 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135510 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:43.135897 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135513 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135516 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135518 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135521 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135524 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135526 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135529 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135531 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135534 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135537 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135539 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135542 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135545 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135547 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135551 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135554 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135557 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135560 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135562 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:43.136375 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135565 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135567 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135570 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135572 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135575 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135577 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135580 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135583 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135585 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135588 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135590 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135593 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135595 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135598 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135601 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135604 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135608 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135612 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135615 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:43.136901 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135617 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135620 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135622 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135625 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135628 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135630 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135633 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135636 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135639 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135642 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135644 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135647 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135649 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135652 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135654 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135659 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135663 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135666 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135668 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135672 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:43.137367 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135674 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135677 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135680 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135683 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135686 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135688 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135691 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135694 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135697 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135700 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135703 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135706 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135708 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135711 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135714 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135717 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135720 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135722 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135726 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135728 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:43.137883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135732 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135735 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135737 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135740 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135743 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135745 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135748 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135750 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135753 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135755 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135758 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135762 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135765 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135768 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135771 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.135773 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136170 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136175 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136178 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136182 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:43.138380 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136185 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136188 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136191 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136194 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136197 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136200 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136202 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136205 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136207 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136210 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136212 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136215 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136218 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136221 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136223 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136226 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136228 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136233 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136236 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:43.138888 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136239 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136242 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136245 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136247 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136250 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136253 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136255 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136258 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136260 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136262 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136265 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136267 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136270 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136273 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136277 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136282 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136285 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136288 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136290 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136293 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:43.139347 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136296 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136299 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136302 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136305 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136308 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136311 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136314 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136317 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136320 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136323 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136326 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136328 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136331 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136333 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136336 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136338 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136341 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136343 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136346 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136348 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:43.139863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136351 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136353 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136356 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136358 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136361 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136363 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136366 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136369 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136372 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136374 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136377 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136379 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136383 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136386 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136389 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136391 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136394 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136396 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136398 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136401 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:43.140334 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136404 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136406 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136408 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136411 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136413 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136416 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136418 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136421 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136423 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136425 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136429 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136432 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136434 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136436 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136439 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.136441 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136553 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136592 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136599 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136604 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136610 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 15 20:36:43.140839 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136613 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136617 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136621 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136625 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136628 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136632 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136635 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136638 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136641 2579 flags.go:64] FLAG: --cgroup-root="" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136644 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136647 2579 flags.go:64] FLAG: --client-ca-file="" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136651 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136654 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136659 2579 flags.go:64] FLAG: --cluster-domain="" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136663 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136666 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136669 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136672 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136676 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136679 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136683 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136686 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136689 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136692 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 15 20:36:43.141360 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136695 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136698 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136701 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136708 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136711 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136714 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136717 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136720 2579 flags.go:64] FLAG: --enable-server="true" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136723 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136727 2579 flags.go:64] FLAG: --event-burst="100" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136730 2579 flags.go:64] FLAG: --event-qps="50" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136733 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136736 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136739 2579 flags.go:64] FLAG: --eviction-hard="" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136743 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136745 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136748 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136751 2579 flags.go:64] FLAG: --eviction-soft="" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136755 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136757 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136763 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136766 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136770 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136773 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136776 2579 flags.go:64] FLAG: --feature-gates="" Sep 15 20:36:43.141970 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136779 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136782 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136786 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136789 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136792 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136795 2579 flags.go:64] FLAG: --help="false" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136798 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136801 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136804 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136807 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136812 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136815 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136818 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136821 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136824 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136827 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136829 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136833 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136836 2579 flags.go:64] FLAG: --kube-reserved="" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136838 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136841 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136844 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136847 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136850 2579 flags.go:64] FLAG: --lock-file="" Sep 15 20:36:43.142726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136852 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136855 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136858 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136864 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136867 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136871 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136874 2579 flags.go:64] FLAG: --logging-format="text" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136877 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136880 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136883 2579 flags.go:64] FLAG: --manifest-url="" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136886 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136891 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136893 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136898 2579 flags.go:64] FLAG: --max-pods="110" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136900 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136904 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136907 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136910 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136915 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136918 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136921 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136929 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136932 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136935 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 15 20:36:43.143321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136938 2579 flags.go:64] FLAG: --pod-cidr="" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136941 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136947 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136950 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136953 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136956 2579 flags.go:64] FLAG: --port="10250" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136959 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136962 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-038c80a36f4187f16" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136965 2579 flags.go:64] FLAG: --qos-reserved="" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136968 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136971 2579 flags.go:64] FLAG: --register-node="true" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136974 2579 flags.go:64] FLAG: --register-with-taints="" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136977 2579 flags.go:64] FLAG: --registry-burst="10" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136982 2579 flags.go:64] FLAG: --registry-qps="5" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136985 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136988 2579 flags.go:64] FLAG: --reserved-memory="" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136991 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136994 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.136997 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137000 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137003 2579 flags.go:64] FLAG: --runonce="false" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137006 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137009 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137012 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137014 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 15 20:36:43.143996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137017 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137020 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137024 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137027 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137030 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137033 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137036 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137039 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137042 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137045 2579 flags.go:64] FLAG: --system-cgroups="" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137047 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137052 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137055 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137058 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137062 2579 flags.go:64] FLAG: --tls-min-version="" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137065 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137068 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137071 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137073 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137077 2579 flags.go:64] FLAG: --v="2" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137082 2579 flags.go:64] FLAG: --version="false" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137087 2579 flags.go:64] FLAG: --vmodule="" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137090 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.137094 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137208 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:43.144630 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137212 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137215 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137218 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137221 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137224 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137226 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137229 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137232 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137234 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137238 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137241 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137243 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137246 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137249 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137251 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137254 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137256 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137259 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137262 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137264 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:43.145455 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137267 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137269 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137272 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137274 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137277 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137279 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137282 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137286 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137288 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137291 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137294 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137297 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137301 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137304 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137307 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137310 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137312 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137315 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137318 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137320 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:43.146326 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137323 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137327 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137330 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137333 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137335 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137338 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137340 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137343 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137346 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137348 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137351 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137354 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137356 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137359 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137361 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137364 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137367 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137369 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137372 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137375 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:43.147252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137378 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137381 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137385 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137387 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137391 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137395 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137398 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137401 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137403 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137406 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137409 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137412 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137415 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137420 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137423 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137425 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137428 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137431 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137433 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:43.148137 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137436 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137439 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137441 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137444 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137446 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137449 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137451 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137454 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137457 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137459 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137462 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137464 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137467 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137478 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137482 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137486 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137490 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137506 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:43.148958 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.137519 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.138458 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.147423 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.147448 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.147468 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.147477 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147584 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147594 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147601 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147606 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147611 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147615 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147620 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147624 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147629 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:43.149483 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147634 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147638 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147643 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147647 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147651 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147656 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147660 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147665 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147669 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147673 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147677 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147681 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147685 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147691 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147695 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147705 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147709 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147714 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147718 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147722 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:43.150100 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147727 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147732 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147736 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147741 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147745 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147750 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147754 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147758 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147762 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147766 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147770 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147775 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147779 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147784 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147788 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147792 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147796 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147800 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147804 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:43.150934 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147809 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147813 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147817 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147821 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147826 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147830 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147835 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147838 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147843 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147848 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147855 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147861 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147866 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147872 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147877 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147881 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147885 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147890 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147894 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147898 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:43.151615 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147903 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147907 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147912 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147916 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147920 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147924 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147929 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147935 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147941 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147946 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147950 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147954 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147958 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147962 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147966 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147970 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147974 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147978 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147983 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147987 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:43.152188 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147991 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.147996 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148000 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148005 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148011 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148017 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148022 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148028 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148032 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148036 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148041 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.148049 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148230 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148239 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:43.152782 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148245 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148249 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148253 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148257 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148262 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148266 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148270 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148275 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148279 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148283 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148288 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148292 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148296 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148300 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148304 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148309 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148313 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148317 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148322 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148326 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:43.153141 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148330 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148335 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148340 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148344 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148349 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148353 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148357 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148362 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148366 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148370 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148374 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148378 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148382 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148386 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148390 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148395 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148402 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148409 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148413 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:43.153702 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148418 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148423 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148428 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148432 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148437 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148441 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148446 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148450 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148454 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148458 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148463 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148467 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148471 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148475 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148479 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148487 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148512 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148518 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148523 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148527 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:43.154156 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148531 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148537 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148541 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148546 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148550 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148554 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148560 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148564 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148568 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148573 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148577 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148582 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148586 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148590 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148594 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148598 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148603 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148607 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148612 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148616 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:43.154775 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148620 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148624 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148629 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148633 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148637 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148641 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148646 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148650 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148655 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148659 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148665 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148670 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148674 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148678 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148682 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148686 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148691 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:43.155291 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:43.148695 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:43.155727 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.148703 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 20:36:43.155727 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.148901 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 15 20:36:43.155727 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.152144 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 15 20:36:43.155727 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.153336 2579 server.go:1019] "Starting client certificate rotation" Sep 15 20:36:43.155727 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.153447 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 20:36:43.155727 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.153486 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 20:36:43.156384 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.156365 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 20:36:43.160712 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.160686 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 20:36:43.170145 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.170126 2579 log.go:25] "Validated CRI v1 runtime API" Sep 15 20:36:43.174700 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.174685 2579 log.go:25] "Validated CRI v1 image API" Sep 15 20:36:43.176319 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.176305 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 15 20:36:43.183100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.183079 2579 fs.go:135] Filesystem UUIDs: map[0be3f607-8cf0-4fe3-9a7c-8785cd2af3d2:/dev/nvme0n1p4 31b330c7-f1a4-4cb2-8eb5-719de8189c7d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 15 20:36:43.183182 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.183099 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 15 20:36:43.184399 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.184382 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 20:36:43.189110 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.188996 2579 manager.go:216] Machine: {Timestamp:2026-09-15 20:36:43.187641311 +0000 UTC m=+0.381594757 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096020 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d86b63c23a2a94cb60deb1744c3d6 SystemUUID:ec2d86b6-3c23-a2a9-4cb6-0deb1744c3d6 BootID:ecbaf48c-ebf6-41f2-b6e2-da096ff2982f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f8:24:7f:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f8:24:7f:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:bd:da:db:4d:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 15 20:36:43.189110 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.189108 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 15 20:36:43.189217 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.189191 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 15 20:36:43.192702 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.192676 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 15 20:36:43.192856 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.192705 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 15 20:36:43.193570 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.193559 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 15 20:36:43.193609 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.193573 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 15 20:36:43.193609 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.193581 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 20:36:43.193609 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.193596 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 20:36:43.194917 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.194905 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 15 20:36:43.199748 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.199731 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 15 20:36:43.199873 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.199863 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 15 20:36:43.199911 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.199899 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 15 20:36:43.199941 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.199916 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 15 20:36:43.199941 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.199932 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 15 20:36:43.200035 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.199945 2579 kubelet.go:398] "Adding apiserver pod source" Sep 15 20:36:43.200035 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.199957 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 15 20:36:43.200306 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.200296 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 20:36:43.200340 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.200320 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 20:36:43.202814 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.202793 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 15 20:36:43.205791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.205776 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 15 20:36:43.205856 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.205807 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 15 20:36:43.207829 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207817 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 15 20:36:43.207880 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207832 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 15 20:36:43.207880 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207838 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 15 20:36:43.207880 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207843 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 15 20:36:43.207880 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207850 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 15 20:36:43.207880 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207855 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 15 20:36:43.207880 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207868 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 15 20:36:43.207880 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207874 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 15 20:36:43.207880 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207881 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 15 20:36:43.208395 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207887 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 15 20:36:43.208395 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207895 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 15 20:36:43.208395 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.207900 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 15 20:36:43.208802 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.208787 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 15 20:36:43.208802 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.208801 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 15 20:36:43.211945 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.211564 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 15 20:36:43.211945 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.211683 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 15 20:36:43.211945 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.211814 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 15 20:36:43.213431 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.213268 2579 server.go:1268] "Started kubelet" Sep 15 20:36:43.213543 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.213343 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 15 20:36:43.213593 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.213359 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 15 20:36:43.213639 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.213623 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 15 20:36:43.214214 ip-10-0-130-107 systemd[1]: Started Kubernetes Kubelet. Sep 15 20:36:43.215380 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.215366 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 15 20:36:43.215473 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.215377 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 15 20:36:43.218556 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.218538 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 15 20:36:43.219090 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.217910 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-107.ec2.internal.18d598f4967387fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-107.ec2.internal,UID:ip-10-0-130-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-107.ec2.internal,},FirstTimestamp:2026-09-15 20:36:43.213236218 +0000 UTC m=+0.407189664,LastTimestamp:2026-09-15 20:36:43.213236218 +0000 UTC m=+0.407189664,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-107.ec2.internal,}" Sep 15 20:36:43.221847 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.221828 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 15 20:36:43.221847 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.221840 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 15 20:36:43.221975 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.221898 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:43.222584 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.222562 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 15 20:36:43.222584 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.222564 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 15 20:36:43.222729 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.222595 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 15 20:36:43.222806 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.222788 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 15 20:36:43.222806 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.222799 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 15 20:36:43.223106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.222897 2579 factory.go:55] Registering systemd factory Sep 15 20:36:43.223106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.222916 2579 factory.go:223] Registration of the systemd container factory successfully Sep 15 20:36:43.223418 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.223394 2579 factory.go:153] Registering CRI-O factory Sep 15 20:36:43.223418 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.223412 2579 factory.go:223] Registration of the crio container factory successfully Sep 15 20:36:43.223578 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.223474 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 15 20:36:43.223578 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.223508 2579 factory.go:103] Registering Raw factory Sep 15 20:36:43.223578 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.223524 2579 manager.go:1196] Started watching for new ooms in manager Sep 15 20:36:43.223907 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.223892 2579 manager.go:349] Starting recovery of all containers Sep 15 20:36:43.228604 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.228428 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zc7dm" Sep 15 20:36:43.230101 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.230056 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 15 20:36:43.230324 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.230299 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 15 20:36:43.230447 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.230426 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 15 20:36:43.234986 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.234964 2579 manager.go:354] Recovery completed Sep 15 20:36:43.235677 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.235655 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zc7dm" Sep 15 20:36:43.240674 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.240555 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:43.242698 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.242680 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:43.242767 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.242713 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:43.242767 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.242727 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:43.243257 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.243243 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 15 20:36:43.243322 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.243259 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 15 20:36:43.243322 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.243277 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 15 20:36:43.243322 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.243260 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:43.244052 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.244039 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:43.244123 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.244068 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:43.244123 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.244083 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:43.245353 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.245339 2579 policy_none.go:49] "None policy: Start" Sep 15 20:36:43.245409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.245357 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 15 20:36:43.245409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.245367 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 15 20:36:43.247394 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.247381 2579 policy_none.go:47] "Start" Sep 15 20:36:43.287090 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.287075 2579 manager.go:337] "Starting Device Plugin manager" Sep 15 20:36:43.287215 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.287112 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 15 20:36:43.287215 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.287122 2579 server.go:87] "Starting device plugin registration server" Sep 15 20:36:43.287487 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.287474 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 15 20:36:43.287551 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.287490 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 15 20:36:43.287651 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.287631 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 15 20:36:43.287767 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.287750 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 15 20:36:43.287881 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.287775 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 15 20:36:43.288452 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.288436 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 15 20:36:43.288559 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.288489 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:43.304706 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.304686 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 15 20:36:43.311930 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.304736 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 15 20:36:43.311930 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.304758 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 15 20:36:43.311930 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.304800 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 15 20:36:43.311930 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.306734 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:43.388255 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.388138 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:43.389204 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.389183 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:43.389328 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.389216 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:43.389328 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.389229 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:43.389328 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.389263 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.398196 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.398166 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.398196 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.398198 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-107.ec2.internal\": node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:43.405003 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.404975 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal"] Sep 15 20:36:43.405144 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.405049 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:43.405977 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.405959 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:43.406090 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.405988 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:43.406090 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.406001 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:43.408187 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.408173 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:43.408305 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.408290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.408340 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.408321 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:43.408993 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.408979 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:43.409084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.409003 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:43.409084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.409016 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:43.409084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.409049 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:43.409084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.409072 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:43.409084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.409086 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:43.411228 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.411209 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.411314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.411246 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:43.412061 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.412041 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:43.412127 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.412073 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:43.412127 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.412083 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:43.422259 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.422242 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:43.433230 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.433210 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-107.ec2.internal\" not found" node="ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.436736 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.436718 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-107.ec2.internal\" not found" node="ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.523028 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.522985 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:43.524122 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.524105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7d376dec3d60bed61757396e0acedf32-config\") pod \"kube-apiserver-proxy-ip-10-0-130-107.ec2.internal\" (UID: \"7d376dec3d60bed61757396e0acedf32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.524182 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.524130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bec34a1f94590650aa2db152c60a6012-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"bec34a1f94590650aa2db152c60a6012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.524182 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.524147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bec34a1f94590650aa2db152c60a6012-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"bec34a1f94590650aa2db152c60a6012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.623410 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.623368 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:43.624539 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.624512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7d376dec3d60bed61757396e0acedf32-config\") pod \"kube-apiserver-proxy-ip-10-0-130-107.ec2.internal\" (UID: \"7d376dec3d60bed61757396e0acedf32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.624657 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.624546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bec34a1f94590650aa2db152c60a6012-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"bec34a1f94590650aa2db152c60a6012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.624657 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.624560 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7d376dec3d60bed61757396e0acedf32-config\") pod \"kube-apiserver-proxy-ip-10-0-130-107.ec2.internal\" (UID: \"7d376dec3d60bed61757396e0acedf32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.624657 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.624584 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bec34a1f94590650aa2db152c60a6012-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"bec34a1f94590650aa2db152c60a6012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.624657 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.624565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bec34a1f94590650aa2db152c60a6012-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"bec34a1f94590650aa2db152c60a6012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.624657 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.624609 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bec34a1f94590650aa2db152c60a6012-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"bec34a1f94590650aa2db152c60a6012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.724089 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.724020 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:43.737261 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.737244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.740449 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:43.740422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" Sep 15 20:36:43.824965 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.824929 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:43.925424 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:43.925400 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:44.025875 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:44.025812 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:44.126540 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:44.126487 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:44.152759 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.152723 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 15 20:36:44.153223 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.152895 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 20:36:44.206225 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.206201 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:44.227630 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:44.227605 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:44.242778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.242746 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 20:31:44 +0000 UTC" deadline="2028-06-09 03:50:10.47442845 +0000 UTC" Sep 15 20:36:44.242778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.242772 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15175h13m26.231659498s" Sep 15 20:36:44.328368 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:44.328305 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:44.373246 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.373231 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 20:36:44.428508 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:44.428469 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:44.500936 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:44.500778 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec34a1f94590650aa2db152c60a6012.slice/crio-dc45ce3040f2273a6c2d07497941ef233ed915b326855340e6e6ae1878bdbac9 WatchSource:0}: Error finding container dc45ce3040f2273a6c2d07497941ef233ed915b326855340e6e6ae1878bdbac9: Status 404 returned error can't find the container with id dc45ce3040f2273a6c2d07497941ef233ed915b326855340e6e6ae1878bdbac9 Sep 15 20:36:44.528775 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:44.528749 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found" Sep 15 20:36:44.610667 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.610609 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:44.622933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.622912 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" Sep 15 20:36:44.636563 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.636547 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 20:36:44.637682 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.637669 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" Sep 15 20:36:44.646017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.646005 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 20:36:44.704027 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:44.703997 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:45.200629 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.200599 2579 apiserver.go:52] "Watching apiserver" Sep 15 20:36:45.210075 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.210042 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 15 20:36:45.210557 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.210535 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-h48b4","kube-system/konnectivity-agent-9mm89","openshift-image-registry/node-ca-895fv","openshift-multus/multus-7qblf","openshift-multus/multus-additional-cni-plugins-n5z2v","openshift-multus/network-metrics-daemon-d628s","openshift-network-diagnostics/network-check-target-5przr","openshift-network-operator/iptables-alerter-dps4j","kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl","openshift-cluster-node-tuning-operator/tuned-bjmhs","openshift-dns/node-resolver-6t5mb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-xmfnc"] Sep 15 20:36:45.213472 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.213389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:45.213567 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.213482 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:36:45.216326 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.215822 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.219310 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.218181 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6cc2t\"" Sep 15 20:36:45.219310 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.218231 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 15 20:36:45.219310 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.218477 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:36:45.219310 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.218511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.220873 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.220612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.221602 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.221120 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 15 20:36:45.221602 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.221138 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 15 20:36:45.221602 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.221285 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4p89d\"" Sep 15 20:36:45.221602 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.221286 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 15 20:36:45.222677 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.222659 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 15 20:36:45.223907 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.223036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.223907 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.223393 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ps6zw\"" Sep 15 20:36:45.223907 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.223427 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 15 20:36:45.223907 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.223440 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 15 20:36:45.223907 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.223538 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 15 20:36:45.223907 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.223738 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 15 20:36:45.225289 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.225270 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 15 20:36:45.225525 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.225510 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 15 20:36:45.225993 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.225972 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vpqft\"" Sep 15 20:36:45.226262 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.226243 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:45.226343 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.226318 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:36:45.228492 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.228433 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.231624 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.230543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.231624 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.230671 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 15 20:36:45.231624 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.230964 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 15 20:36:45.231624 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.231295 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-69mt2\"" Sep 15 20:36:45.231624 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.231443 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 15 20:36:45.231624 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.231493 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 15 20:36:45.231624 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.231527 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 15 20:36:45.231969 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.231791 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 15 20:36:45.232938 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.232916 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:36:45.233364 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.233344 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wmvl2\"" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e29a808c-8900-425b-b41b-902affb6253c-cni-binary-copy\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-hostroot\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e29a808c-8900-425b-b41b-902affb6253c-multus-daemon-config\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-cni-binary-copy\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-multus-cni-dir\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234850 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-var-lib-kubelet\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234872 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmd7w\" (UniqueName: \"kubernetes.io/projected/e29a808c-8900-425b-b41b-902affb6253c-kube-api-access-mmd7w\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-run\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-sys\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234953 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-etc-kubernetes\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.234995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.234997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-sysctl-d\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235023 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-lib-modules\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-cnibin\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-multus-socket-dir-parent\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-modprobe-d\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ee2883a-2e3b-4546-8bb7-81d068335229-tmp\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn929\" (UniqueName: \"kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929\") pod \"network-check-target-5przr\" (UID: \"4804eb98-92c2-4976-a35c-e4079e799832\") " pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-systemd\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8md5v\" (UniqueName: \"kubernetes.io/projected/6ee2883a-2e3b-4546-8bb7-81d068335229-kube-api-access-8md5v\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-system-cni-dir\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-var-lib-cni-bin\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-system-cni-dir\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hn5\" (UniqueName: \"kubernetes.io/projected/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-kube-api-access-g5hn5\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-var-lib-kubelet\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.235697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-tuned\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c22bdf6e-dac6-4791-b78d-f713a85241ea-host\") pod \"node-ca-895fv\" (UID: \"c22bdf6e-dac6-4791-b78d-f713a85241ea\") " pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4hcq\" (UniqueName: \"kubernetes.io/projected/c22bdf6e-dac6-4791-b78d-f713a85241ea-kube-api-access-p4hcq\") pod \"node-ca-895fv\" (UID: \"c22bdf6e-dac6-4791-b78d-f713a85241ea\") " pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-run-k8s-cni-cncf-io\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-var-lib-cni-multus\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-multus-conf-dir\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-cnibin\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-kubernetes\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235807 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-sysctl-conf\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-run-netns\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-run-multus-certs\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-sysconfig\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-host\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.236519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.235958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c22bdf6e-dac6-4791-b78d-f713a85241ea-serviceca\") pod \"node-ca-895fv\" (UID: \"c22bdf6e-dac6-4791-b78d-f713a85241ea\") " pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.237506 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.237484 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 20:36:45.239620 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.239601 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.247470 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.247451 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6zwwd\"" Sep 15 20:36:45.247741 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.247686 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 15 20:36:45.248065 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.248018 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 15 20:36:45.249099 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.249079 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:45.249610 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.249586 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nsrfz\"" Sep 15 20:36:45.249804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.249788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 15 20:36:45.249877 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.249822 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 15 20:36:45.249877 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.249867 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 15 20:36:45.250129 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.250106 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:36:45.251125 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.251104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.253439 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.253168 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 15 20:36:45.253509 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.253483 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kqk2l\"" Sep 15 20:36:45.253717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.253696 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 15 20:36:45.262371 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.262349 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4cxh2" Sep 15 20:36:45.270829 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.270810 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4cxh2" Sep 15 20:36:45.312930 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.312877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" event={"ID":"bec34a1f94590650aa2db152c60a6012","Type":"ContainerStarted","Data":"dc45ce3040f2273a6c2d07497941ef233ed915b326855340e6e6ae1878bdbac9"} Sep 15 20:36:45.315158 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.315130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" event={"ID":"7d376dec3d60bed61757396e0acedf32","Type":"ContainerStarted","Data":"da1d5a7562d99771abb1acab6ad82cab8441df103e7b7d4e9fc5df2390663dc3"} Sep 15 20:36:45.324329 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.324303 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 15 20:36:45.336309 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-sysconfig\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.336309 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e29a808c-8900-425b-b41b-902affb6253c-cni-binary-copy\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.336567 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-hostroot\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.336567 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e29a808c-8900-425b-b41b-902affb6253c-multus-daemon-config\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.336567 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-cni-binary-copy\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.336567 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-sysconfig\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.336567 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-slash\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.336567 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336431 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-hostroot\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.336567 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-cni-bin\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.336567 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28a9151d-ae36-4a20-bd50-ab9db08bde8f-ovn-node-metrics-cert\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmd7w\" (UniqueName: \"kubernetes.io/projected/e29a808c-8900-425b-b41b-902affb6253c-kube-api-access-mmd7w\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-run-netns\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-cni-netd\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-socket-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-registration-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-sys\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-etc-kubernetes\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28a9151d-ae36-4a20-bd50-ab9db08bde8f-ovnkube-config\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336828 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4e3d5cf-fff4-45f3-b820-70c6baee1155-konnectivity-ca\") pod \"konnectivity-agent-9mm89\" (UID: \"d4e3d5cf-fff4-45f3-b820-70c6baee1155\") " pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa34b802-6f15-48e4-bf1b-fb01f9c2feb3-tmp-dir\") pod \"node-resolver-6t5mb\" (UID: \"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3\") " pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm2rv\" (UniqueName: \"kubernetes.io/projected/fa34b802-6f15-48e4-bf1b-fb01f9c2feb3-kube-api-access-lm2rv\") pod \"node-resolver-6t5mb\" (UID: \"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3\") " pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-lib-modules\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-cnibin\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.336958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-multus-socket-dir-parent\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.336993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa34b802-6f15-48e4-bf1b-fb01f9c2feb3-hosts-file\") pod \"node-resolver-6t5mb\" (UID: \"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3\") " pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ee2883a-2e3b-4546-8bb7-81d068335229-tmp\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28a9151d-ae36-4a20-bd50-ab9db08bde8f-ovnkube-script-lib\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn929\" (UniqueName: \"kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929\") pod \"network-check-target-5przr\" (UID: \"4804eb98-92c2-4976-a35c-e4079e799832\") " pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-var-lib-cni-bin\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-system-cni-dir\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337093 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-cni-binary-copy\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337094 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e29a808c-8900-425b-b41b-902affb6253c-multus-daemon-config\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e29a808c-8900-425b-b41b-902affb6253c-cni-binary-copy\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqr24\" (UniqueName: \"kubernetes.io/projected/28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0-kube-api-access-tqr24\") pod \"iptables-alerter-dps4j\" (UID: \"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0\") " pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-sys\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-lib-modules\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-etc-kubernetes\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-system-cni-dir\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.337813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337427 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-var-lib-cni-bin\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4e3d5cf-fff4-45f3-b820-70c6baee1155-agent-certs\") pod \"konnectivity-agent-9mm89\" (UID: \"d4e3d5cf-fff4-45f3-b820-70c6baee1155\") " pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337474 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-multus-socket-dir-parent\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-var-lib-kubelet\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-var-lib-cni-multus\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337538 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-var-lib-kubelet\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337540 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-cnibin\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-cnibin\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-var-lib-cni-multus\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337680 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-cnibin\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-sys-fs\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-kubernetes\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337769 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337786 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-kubernetes\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-sysctl-conf\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-run-netns\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.338747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-run-netns\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-host\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c22bdf6e-dac6-4791-b78d-f713a85241ea-serviceca\") pod \"node-ca-895fv\" (UID: \"c22bdf6e-dac6-4791-b78d-f713a85241ea\") " pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x682c\" (UniqueName: \"kubernetes.io/projected/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-kube-api-access-x682c\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-sysctl-conf\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.337970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-multus-cni-dir\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-host\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-var-lib-kubelet\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd252\" (UniqueName: \"kubernetes.io/projected/11e01e0e-b7c2-41fb-8152-77fdd5533940-kube-api-access-fd252\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0-host-slash\") pod \"iptables-alerter-dps4j\" (UID: \"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0\") " pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-run\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-log-socket\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-sysctl-d\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-run-openvswitch\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-run-ovn\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdfvq\" (UniqueName: \"kubernetes.io/projected/28a9151d-ae36-4a20-bd50-ab9db08bde8f-kube-api-access-gdfvq\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.339640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-modprobe-d\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-var-lib-openvswitch\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338426 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-multus-cni-dir\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-sysctl-d\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-etc-openvswitch\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-modprobe-d\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-node-log\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/609fe877-8116-4e94-a791-ab35d08392e1-kubelet-config\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338563 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-var-lib-kubelet\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-systemd-units\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-systemd\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8md5v\" (UniqueName: \"kubernetes.io/projected/6ee2883a-2e3b-4546-8bb7-81d068335229-kube-api-access-8md5v\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-system-cni-dir\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hn5\" (UniqueName: \"kubernetes.io/projected/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-kube-api-access-g5hn5\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.340314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-kubelet\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28a9151d-ae36-4a20-bd50-ab9db08bde8f-env-overrides\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0-iptables-alerter-script\") pod \"iptables-alerter-dps4j\" (UID: \"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0\") " pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-tuned\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c22bdf6e-dac6-4791-b78d-f713a85241ea-host\") pod \"node-ca-895fv\" (UID: \"c22bdf6e-dac6-4791-b78d-f713a85241ea\") " pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4hcq\" (UniqueName: \"kubernetes.io/projected/c22bdf6e-dac6-4791-b78d-f713a85241ea-kube-api-access-p4hcq\") pod \"node-ca-895fv\" (UID: \"c22bdf6e-dac6-4791-b78d-f713a85241ea\") " pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.338975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-run-k8s-cni-cncf-io\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-multus-conf-dir\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-run-systemd\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-device-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-run-multus-certs\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c22bdf6e-dac6-4791-b78d-f713a85241ea-serviceca\") pod \"node-ca-895fv\" (UID: \"c22bdf6e-dac6-4791-b78d-f713a85241ea\") " pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/609fe877-8116-4e94-a791-ab35d08392e1-dbus\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339052 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-run\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-run-multus-certs\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.340882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339267 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-host-run-k8s-cni-cncf-io\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.341433 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-multus-conf-dir\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.341433 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-systemd\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.341433 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339336 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c22bdf6e-dac6-4791-b78d-f713a85241ea-host\") pod \"node-ca-895fv\" (UID: \"c22bdf6e-dac6-4791-b78d-f713a85241ea\") " pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.341433 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.339468 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e29a808c-8900-425b-b41b-902affb6253c-system-cni-dir\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.342510 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.342462 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ee2883a-2e3b-4546-8bb7-81d068335229-tmp\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.342510 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.342477 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ee2883a-2e3b-4546-8bb7-81d068335229-etc-tuned\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.350837 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.350762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hn5\" (UniqueName: \"kubernetes.io/projected/0309b1b6-8ef8-4242-a7b0-dcd5bccb9694-kube-api-access-g5hn5\") pod \"multus-additional-cni-plugins-n5z2v\" (UID: \"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694\") " pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.351168 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.351143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmd7w\" (UniqueName: \"kubernetes.io/projected/e29a808c-8900-425b-b41b-902affb6253c-kube-api-access-mmd7w\") pod \"multus-7qblf\" (UID: \"e29a808c-8900-425b-b41b-902affb6253c\") " pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.351405 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.351329 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:36:45.351405 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.351347 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:36:45.351405 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.351358 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fn929 for pod openshift-network-diagnostics/network-check-target-5przr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:45.351664 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.351430 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929 podName:4804eb98-92c2-4976-a35c-e4079e799832 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:45.851399174 +0000 UTC m=+3.045352608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fn929" (UniqueName: "kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929") pod "network-check-target-5przr" (UID: "4804eb98-92c2-4976-a35c-e4079e799832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:45.354777 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.354496 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8md5v\" (UniqueName: \"kubernetes.io/projected/6ee2883a-2e3b-4546-8bb7-81d068335229-kube-api-access-8md5v\") pod \"tuned-bjmhs\" (UID: \"6ee2883a-2e3b-4546-8bb7-81d068335229\") " pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.355437 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.355414 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4hcq\" (UniqueName: \"kubernetes.io/projected/c22bdf6e-dac6-4791-b78d-f713a85241ea-kube-api-access-p4hcq\") pod \"node-ca-895fv\" (UID: \"c22bdf6e-dac6-4791-b78d-f713a85241ea\") " pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.439457 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:45.439457 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqr24\" (UniqueName: \"kubernetes.io/projected/28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0-kube-api-access-tqr24\") pod \"iptables-alerter-dps4j\" (UID: \"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0\") " pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.439730 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4e3d5cf-fff4-45f3-b820-70c6baee1155-agent-certs\") pod \"konnectivity-agent-9mm89\" (UID: \"d4e3d5cf-fff4-45f3-b820-70c6baee1155\") " pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:36:45.439730 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.439554 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:45.439730 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.439636 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret podName:609fe877-8116-4e94-a791-ab35d08392e1 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:45.939612056 +0000 UTC m=+3.133565492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret") pod "global-pull-secret-syncer-h48b4" (UID: "609fe877-8116-4e94-a791-ab35d08392e1") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:45.439730 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.439964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439757 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.439964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-sys-fs\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.439964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.439964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x682c\" (UniqueName: \"kubernetes.io/projected/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-kube-api-access-x682c\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.439964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-sys-fs\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.439964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd252\" (UniqueName: \"kubernetes.io/projected/11e01e0e-b7c2-41fb-8152-77fdd5533940-kube-api-access-fd252\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:45.439964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.439933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0-host-slash\") pod \"iptables-alerter-dps4j\" (UID: \"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0\") " pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-log-socket\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-run-openvswitch\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440166 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0-host-slash\") pod \"iptables-alerter-dps4j\" (UID: \"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0\") " pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-run-ovn\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdfvq\" (UniqueName: \"kubernetes.io/projected/28a9151d-ae36-4a20-bd50-ab9db08bde8f-kube-api-access-gdfvq\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440238 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-run-openvswitch\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440235 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-log-socket\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-var-lib-openvswitch\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440272 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-run-ovn\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-etc-openvswitch\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-node-log\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/609fe877-8116-4e94-a791-ab35d08392e1-kubelet-config\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440334 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-var-lib-openvswitch\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-systemd-units\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440385 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-etc-openvswitch\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-node-log\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440405 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-systemd-units\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440417 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/609fe877-8116-4e94-a791-ab35d08392e1-kubelet-config\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-kubelet\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28a9151d-ae36-4a20-bd50-ab9db08bde8f-env-overrides\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0-iptables-alerter-script\") pod \"iptables-alerter-dps4j\" (UID: \"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0\") " pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-kubelet\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-run-systemd\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-device-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440612 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-device-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440653 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-run-systemd\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/609fe877-8116-4e94-a791-ab35d08392e1-dbus\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:45.440859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-slash\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-cni-bin\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28a9151d-ae36-4a20-bd50-ab9db08bde8f-ovn-node-metrics-cert\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-run-netns\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-slash\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440833 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-cni-bin\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-cni-netd\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440908 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/609fe877-8116-4e94-a791-ab35d08392e1-dbus\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-cni-netd\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-socket-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-registration-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28a9151d-ae36-4a20-bd50-ab9db08bde8f-env-overrides\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.440995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28a9151d-ae36-4a20-bd50-ab9db08bde8f-ovnkube-config\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4e3d5cf-fff4-45f3-b820-70c6baee1155-konnectivity-ca\") pod \"konnectivity-agent-9mm89\" (UID: \"d4e3d5cf-fff4-45f3-b820-70c6baee1155\") " pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441106 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.441635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa34b802-6f15-48e4-bf1b-fb01f9c2feb3-tmp-dir\") pod \"node-resolver-6t5mb\" (UID: \"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3\") " pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm2rv\" (UniqueName: \"kubernetes.io/projected/fa34b802-6f15-48e4-bf1b-fb01f9c2feb3-kube-api-access-lm2rv\") pod \"node-resolver-6t5mb\" (UID: \"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3\") " pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-registration-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa34b802-6f15-48e4-bf1b-fb01f9c2feb3-hosts-file\") pod \"node-resolver-6t5mb\" (UID: \"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3\") " pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441196 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0-iptables-alerter-script\") pod \"iptables-alerter-dps4j\" (UID: \"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0\") " pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28a9151d-ae36-4a20-bd50-ab9db08bde8f-ovnkube-script-lib\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28a9151d-ae36-4a20-bd50-ab9db08bde8f-host-run-netns\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-socket-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.441354 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.441405 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs podName:11e01e0e-b7c2-41fb-8152-77fdd5533940 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:45.941387296 +0000 UTC m=+3.135340752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs") pod "network-metrics-daemon-d628s" (UID: "11e01e0e-b7c2-41fb-8152-77fdd5533940") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441470 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441523 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa34b802-6f15-48e4-bf1b-fb01f9c2feb3-hosts-file\") pod \"node-resolver-6t5mb\" (UID: \"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3\") " pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4e3d5cf-fff4-45f3-b820-70c6baee1155-konnectivity-ca\") pod \"konnectivity-agent-9mm89\" (UID: \"d4e3d5cf-fff4-45f3-b820-70c6baee1155\") " pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28a9151d-ae36-4a20-bd50-ab9db08bde8f-ovnkube-config\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.441965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa34b802-6f15-48e4-bf1b-fb01f9c2feb3-tmp-dir\") pod \"node-resolver-6t5mb\" (UID: \"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3\") " pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.442409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.442088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28a9151d-ae36-4a20-bd50-ab9db08bde8f-ovnkube-script-lib\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.443088 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.442837 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4e3d5cf-fff4-45f3-b820-70c6baee1155-agent-certs\") pod \"konnectivity-agent-9mm89\" (UID: \"d4e3d5cf-fff4-45f3-b820-70c6baee1155\") " pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:36:45.443552 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.443524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28a9151d-ae36-4a20-bd50-ab9db08bde8f-ovn-node-metrics-cert\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.450037 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.450008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqr24\" (UniqueName: \"kubernetes.io/projected/28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0-kube-api-access-tqr24\") pod \"iptables-alerter-dps4j\" (UID: \"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0\") " pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.450924 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.450861 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdfvq\" (UniqueName: \"kubernetes.io/projected/28a9151d-ae36-4a20-bd50-ab9db08bde8f-kube-api-access-gdfvq\") pod \"ovnkube-node-xmfnc\" (UID: \"28a9151d-ae36-4a20-bd50-ab9db08bde8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.450924 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.450890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd252\" (UniqueName: \"kubernetes.io/projected/11e01e0e-b7c2-41fb-8152-77fdd5533940-kube-api-access-fd252\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:45.451411 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.451206 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x682c\" (UniqueName: \"kubernetes.io/projected/bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5-kube-api-access-x682c\") pod \"aws-ebs-csi-driver-node-dm2hl\" (UID: \"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.451411 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.451340 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm2rv\" (UniqueName: \"kubernetes.io/projected/fa34b802-6f15-48e4-bf1b-fb01f9c2feb3-kube-api-access-lm2rv\") pod \"node-resolver-6t5mb\" (UID: \"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3\") " pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.545180 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.545142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" Sep 15 20:36:45.557273 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.557242 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-895fv" Sep 15 20:36:45.566182 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.566160 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qblf" Sep 15 20:36:45.575207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.575181 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" Sep 15 20:36:45.582384 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.582345 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:36:45.592488 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.592457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dps4j" Sep 15 20:36:45.603729 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.603697 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:36:45.613069 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.612864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" Sep 15 20:36:45.620215 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.620188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6t5mb" Sep 15 20:36:45.695961 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.695929 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:45.945858 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.945771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:45.945858 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.945815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn929\" (UniqueName: \"kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929\") pod \"network-check-target-5przr\" (UID: \"4804eb98-92c2-4976-a35c-e4079e799832\") " pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:45.945858 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:45.945836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:45.946088 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.945952 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:45.946088 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.945953 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:45.946088 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.945984 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:36:45.946088 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.946012 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:36:45.946088 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.946014 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret podName:609fe877-8116-4e94-a791-ab35d08392e1 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:46.94599775 +0000 UTC m=+4.139951189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret") pod "global-pull-secret-syncer-h48b4" (UID: "609fe877-8116-4e94-a791-ab35d08392e1") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:45.946088 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.946027 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fn929 for pod openshift-network-diagnostics/network-check-target-5przr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:45.946088 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.946037 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs podName:11e01e0e-b7c2-41fb-8152-77fdd5533940 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:46.946030218 +0000 UTC m=+4.139983653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs") pod "network-metrics-daemon-d628s" (UID: "11e01e0e-b7c2-41fb-8152-77fdd5533940") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:45.946088 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:45.946082 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929 podName:4804eb98-92c2-4976-a35c-e4079e799832 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:46.946063543 +0000 UTC m=+4.140016996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn929" (UniqueName: "kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929") pod "network-check-target-5przr" (UID: "4804eb98-92c2-4976-a35c-e4079e799832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:46.275178 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:46.275082 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 20:31:46 +0000 UTC" deadline="2028-06-02 00:18:53.358572308 +0000 UTC" Sep 15 20:36:46.275178 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:46.275114 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15003h42m7.083461672s" Sep 15 20:36:46.491048 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:46.491017 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29a808c_8900_425b_b41b_902affb6253c.slice/crio-d0bf173a7ae61e95137782ce33a29ebfb27c212f7dcabc18e26c9820d4cbe542 WatchSource:0}: Error finding container d0bf173a7ae61e95137782ce33a29ebfb27c212f7dcabc18e26c9820d4cbe542: Status 404 returned error can't find the container with id d0bf173a7ae61e95137782ce33a29ebfb27c212f7dcabc18e26c9820d4cbe542 Sep 15 20:36:46.492139 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:46.492107 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a4cb6e_3d5d_4082_b1d8_f4aa74b2a6a0.slice/crio-a000ea4f97f6bcd30de0134ae1c3061220670a1a6aa8a61ca6fa7f35845a64f9 WatchSource:0}: Error finding container a000ea4f97f6bcd30de0134ae1c3061220670a1a6aa8a61ca6fa7f35845a64f9: Status 404 returned error can't find the container with id a000ea4f97f6bcd30de0134ae1c3061220670a1a6aa8a61ca6fa7f35845a64f9 Sep 15 20:36:46.493052 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:46.492926 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba9a9cf_64f8_49fe_bbf1_f9fd858fe9c5.slice/crio-068685d40e783c8c8f8b43248e5009debd65af9f406446a6213ee080ee59da7b WatchSource:0}: Error finding container 068685d40e783c8c8f8b43248e5009debd65af9f406446a6213ee080ee59da7b: Status 404 returned error can't find the container with id 068685d40e783c8c8f8b43248e5009debd65af9f406446a6213ee080ee59da7b Sep 15 20:36:46.496123 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:46.495952 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e3d5cf_fff4_45f3_b820_70c6baee1155.slice/crio-0137070aa66d86c1744c44131def9dcafddcabfb26c05fc9504152b9222d7de7 WatchSource:0}: Error finding container 0137070aa66d86c1744c44131def9dcafddcabfb26c05fc9504152b9222d7de7: Status 404 returned error can't find the container with id 0137070aa66d86c1744c44131def9dcafddcabfb26c05fc9504152b9222d7de7 Sep 15 20:36:46.496305 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:46.496284 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0309b1b6_8ef8_4242_a7b0_dcd5bccb9694.slice/crio-1e0e1eeaee8271ac3e19881c6c7c3a92c55d8d6a5575b737c3a26476eda04f07 WatchSource:0}: Error finding container 1e0e1eeaee8271ac3e19881c6c7c3a92c55d8d6a5575b737c3a26476eda04f07: Status 404 returned error can't find the container with id 1e0e1eeaee8271ac3e19881c6c7c3a92c55d8d6a5575b737c3a26476eda04f07 Sep 15 20:36:46.498306 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:46.498285 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22bdf6e_dac6_4791_b78d_f713a85241ea.slice/crio-c14ffa13941e51b9d2e1955821fdd17f215674ac559dad2562440a05d8ef997a WatchSource:0}: Error finding container c14ffa13941e51b9d2e1955821fdd17f215674ac559dad2562440a05d8ef997a: Status 404 returned error can't find the container with id c14ffa13941e51b9d2e1955821fdd17f215674ac559dad2562440a05d8ef997a Sep 15 20:36:46.499723 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:46.499703 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a9151d_ae36_4a20_bd50_ab9db08bde8f.slice/crio-2f09ddfa5bceab21ee4d62aaad6baf808b57e1fe8ccc09872c92cff9b6828baa WatchSource:0}: Error finding container 2f09ddfa5bceab21ee4d62aaad6baf808b57e1fe8ccc09872c92cff9b6828baa: Status 404 returned error can't find the container with id 2f09ddfa5bceab21ee4d62aaad6baf808b57e1fe8ccc09872c92cff9b6828baa Sep 15 20:36:46.501579 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:46.501392 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa34b802_6f15_48e4_bf1b_fb01f9c2feb3.slice/crio-5348e569f51af2d05eb97f9acd27c81b68e4f8c54cd58548b77680c3f7f36e91 WatchSource:0}: Error finding container 5348e569f51af2d05eb97f9acd27c81b68e4f8c54cd58548b77680c3f7f36e91: Status 404 returned error can't find the container with id 5348e569f51af2d05eb97f9acd27c81b68e4f8c54cd58548b77680c3f7f36e91 Sep 15 20:36:46.504009 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:36:46.503969 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee2883a_2e3b_4546_8bb7_81d068335229.slice/crio-4d4186f1956470ad43b1517037d344c97391274b4ac8a76be74d43c54a2308fa WatchSource:0}: Error finding container 4d4186f1956470ad43b1517037d344c97391274b4ac8a76be74d43c54a2308fa: Status 404 returned error can't find the container with id 4d4186f1956470ad43b1517037d344c97391274b4ac8a76be74d43c54a2308fa Sep 15 20:36:46.953995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:46.953950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn929\" (UniqueName: \"kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929\") pod \"network-check-target-5przr\" (UID: \"4804eb98-92c2-4976-a35c-e4079e799832\") " pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:46.954180 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:46.954010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:46.954180 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:46.954086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:46.954309 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:46.954192 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:36:46.954309 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:46.954224 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:36:46.954309 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:46.954239 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fn929 for pod openshift-network-diagnostics/network-check-target-5przr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:46.954309 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:46.954249 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:46.954309 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:46.954195 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:46.954309 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:46.954304 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929 podName:4804eb98-92c2-4976-a35c-e4079e799832 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:48.954283551 +0000 UTC m=+6.148236991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn929" (UniqueName: "kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929") pod "network-check-target-5przr" (UID: "4804eb98-92c2-4976-a35c-e4079e799832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:46.954831 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:46.954332 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs podName:11e01e0e-b7c2-41fb-8152-77fdd5533940 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:48.954313564 +0000 UTC m=+6.148267009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs") pod "network-metrics-daemon-d628s" (UID: "11e01e0e-b7c2-41fb-8152-77fdd5533940") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:46.954831 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:46.954350 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret podName:609fe877-8116-4e94-a791-ab35d08392e1 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:48.95433996 +0000 UTC m=+6.148293396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret") pod "global-pull-secret-syncer-h48b4" (UID: "609fe877-8116-4e94-a791-ab35d08392e1") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:47.275662 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.275528 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 20:31:46 +0000 UTC" deadline="2028-04-28 09:27:42.425897024 +0000 UTC" Sep 15 20:36:47.275662 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.275596 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14172h50m55.150305205s" Sep 15 20:36:47.308439 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.308402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:47.308670 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:47.308640 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:36:47.309108 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.309084 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:47.309206 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:47.309186 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:36:47.309267 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.309256 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:47.309368 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:47.309347 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:36:47.339993 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.338186 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" event={"ID":"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694","Type":"ContainerStarted","Data":"1e0e1eeaee8271ac3e19881c6c7c3a92c55d8d6a5575b737c3a26476eda04f07"} Sep 15 20:36:47.340177 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.340023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" event={"ID":"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5","Type":"ContainerStarted","Data":"068685d40e783c8c8f8b43248e5009debd65af9f406446a6213ee080ee59da7b"} Sep 15 20:36:47.343597 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.343554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dps4j" event={"ID":"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0","Type":"ContainerStarted","Data":"a000ea4f97f6bcd30de0134ae1c3061220670a1a6aa8a61ca6fa7f35845a64f9"} Sep 15 20:36:47.347807 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.347778 2579 generic.go:356] "Generic (PLEG): container finished" podID="bec34a1f94590650aa2db152c60a6012" containerID="89923c0bafdfdc39746d0dc620a40aed949da6106f055d7f3d7b43bae36e353f" exitCode=0 Sep 15 20:36:47.347945 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.347864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" event={"ID":"bec34a1f94590650aa2db152c60a6012","Type":"ContainerDied","Data":"89923c0bafdfdc39746d0dc620a40aed949da6106f055d7f3d7b43bae36e353f"} Sep 15 20:36:47.352485 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.352429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" event={"ID":"7d376dec3d60bed61757396e0acedf32","Type":"ContainerStarted","Data":"7b2b200d6854767647d054b577c807b51e3e7a46032e1283fabd0df39c8f2ecc"} Sep 15 20:36:47.356690 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.356652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" event={"ID":"6ee2883a-2e3b-4546-8bb7-81d068335229","Type":"ContainerStarted","Data":"4d4186f1956470ad43b1517037d344c97391274b4ac8a76be74d43c54a2308fa"} Sep 15 20:36:47.362604 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.358650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6t5mb" event={"ID":"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3","Type":"ContainerStarted","Data":"5348e569f51af2d05eb97f9acd27c81b68e4f8c54cd58548b77680c3f7f36e91"} Sep 15 20:36:47.366015 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.365986 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" event={"ID":"28a9151d-ae36-4a20-bd50-ab9db08bde8f","Type":"ContainerStarted","Data":"2f09ddfa5bceab21ee4d62aaad6baf808b57e1fe8ccc09872c92cff9b6828baa"} Sep 15 20:36:47.382417 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.382275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-895fv" event={"ID":"c22bdf6e-dac6-4791-b78d-f713a85241ea","Type":"ContainerStarted","Data":"c14ffa13941e51b9d2e1955821fdd17f215674ac559dad2562440a05d8ef997a"} Sep 15 20:36:47.392597 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.387413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9mm89" event={"ID":"d4e3d5cf-fff4-45f3-b820-70c6baee1155","Type":"ContainerStarted","Data":"0137070aa66d86c1744c44131def9dcafddcabfb26c05fc9504152b9222d7de7"} Sep 15 20:36:47.397228 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:47.397194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qblf" event={"ID":"e29a808c-8900-425b-b41b-902affb6253c","Type":"ContainerStarted","Data":"d0bf173a7ae61e95137782ce33a29ebfb27c212f7dcabc18e26c9820d4cbe542"} Sep 15 20:36:48.409429 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:48.409149 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" event={"ID":"bec34a1f94590650aa2db152c60a6012","Type":"ContainerStarted","Data":"0ef6fd3019c988c0380f76b85722908640497d404fc3222b2283bd9be93e1e82"} Sep 15 20:36:48.425778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:48.425716 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" podStartSLOduration=3.425692744 podStartE2EDuration="3.425692744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:36:47.378066381 +0000 UTC m=+4.572019835" watchObservedRunningTime="2026-09-15 20:36:48.425692744 +0000 UTC m=+5.619646204" Sep 15 20:36:48.976634 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:48.976592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:48.976842 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:48.976649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn929\" (UniqueName: \"kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929\") pod \"network-check-target-5przr\" (UID: \"4804eb98-92c2-4976-a35c-e4079e799832\") " pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:48.976842 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:48.976680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:48.976963 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:48.976857 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:48.976963 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:48.976929 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret podName:609fe877-8116-4e94-a791-ab35d08392e1 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:52.976907713 +0000 UTC m=+10.170861164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret") pod "global-pull-secret-syncer-h48b4" (UID: "609fe877-8116-4e94-a791-ab35d08392e1") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:48.977379 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:48.977359 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:48.977460 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:48.977420 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs podName:11e01e0e-b7c2-41fb-8152-77fdd5533940 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:52.977403529 +0000 UTC m=+10.171356965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs") pod "network-metrics-daemon-d628s" (UID: "11e01e0e-b7c2-41fb-8152-77fdd5533940") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:48.977530 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:48.977520 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:36:48.977610 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:48.977539 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:36:48.977610 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:48.977553 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fn929 for pod openshift-network-diagnostics/network-check-target-5przr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:48.977723 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:48.977615 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929 podName:4804eb98-92c2-4976-a35c-e4079e799832 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:52.977598158 +0000 UTC m=+10.171551611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn929" (UniqueName: "kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929") pod "network-check-target-5przr" (UID: "4804eb98-92c2-4976-a35c-e4079e799832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:49.307549 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:49.307462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:49.307746 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:49.307618 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:36:49.308018 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:49.307996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:49.308146 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:49.308107 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:36:49.308146 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:49.308157 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:49.308295 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:49.308222 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:36:51.305749 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:51.305704 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:51.306244 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:51.305857 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:36:51.306390 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:51.306368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:51.306513 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:51.306490 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:36:51.306604 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:51.306563 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:51.306695 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:51.306668 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:36:53.015009 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:53.014966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:53.015495 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:53.015029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn929\" (UniqueName: \"kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929\") pod \"network-check-target-5przr\" (UID: \"4804eb98-92c2-4976-a35c-e4079e799832\") " pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:53.015495 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:53.015063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:53.015495 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.015208 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:53.015495 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.015276 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret podName:609fe877-8116-4e94-a791-ab35d08392e1 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:01.015255067 +0000 UTC m=+18.209208507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret") pod "global-pull-secret-syncer-h48b4" (UID: "609fe877-8116-4e94-a791-ab35d08392e1") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:53.015781 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.015697 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:53.015781 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.015748 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs podName:11e01e0e-b7c2-41fb-8152-77fdd5533940 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:01.015735503 +0000 UTC m=+18.209688939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs") pod "network-metrics-daemon-d628s" (UID: "11e01e0e-b7c2-41fb-8152-77fdd5533940") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:53.015862 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.015827 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:36:53.015862 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.015838 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:36:53.015862 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.015847 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fn929 for pod openshift-network-diagnostics/network-check-target-5przr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:53.015967 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.015881 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929 podName:4804eb98-92c2-4976-a35c-e4079e799832 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:01.015869795 +0000 UTC m=+18.209823231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn929" (UniqueName: "kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929") pod "network-check-target-5przr" (UID: "4804eb98-92c2-4976-a35c-e4079e799832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:53.309098 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:53.309021 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:53.309720 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:53.309307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:53.309720 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.309581 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:36:53.309720 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.309407 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:36:53.309720 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:53.309309 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:53.309720 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:53.309691 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:36:55.308203 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:55.307425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:55.308203 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:55.307588 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:36:55.308203 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:55.307669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:55.308203 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:55.307740 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:36:55.308203 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:55.307782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:55.308203 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:55.307845 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:36:57.305961 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:57.305932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:57.306386 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:57.306060 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:36:57.306529 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:57.306512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:57.306644 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:57.306622 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:36:57.306769 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:57.306743 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:57.306840 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:57.306817 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:36:59.306404 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:59.305882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:36:59.306404 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:59.305940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:36:59.306404 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:36:59.305882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:36:59.306404 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:59.306026 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:36:59.306404 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:59.306133 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:36:59.306404 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:36:59.306285 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:37:01.080742 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:01.080698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:01.080742 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:01.080741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn929\" (UniqueName: \"kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929\") pod \"network-check-target-5przr\" (UID: \"4804eb98-92c2-4976-a35c-e4079e799832\") " pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:01.081239 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:01.080759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:01.081239 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.080858 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:37:01.081239 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.080859 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:37:01.081239 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.080894 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:37:01.081239 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.080913 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs podName:11e01e0e-b7c2-41fb-8152-77fdd5533940 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:17.080897997 +0000 UTC m=+34.274851430 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs") pod "network-metrics-daemon-d628s" (UID: "11e01e0e-b7c2-41fb-8152-77fdd5533940") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:37:01.081239 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.080921 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:37:01.081239 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.080928 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret podName:609fe877-8116-4e94-a791-ab35d08392e1 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:17.080921182 +0000 UTC m=+34.274874615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret") pod "global-pull-secret-syncer-h48b4" (UID: "609fe877-8116-4e94-a791-ab35d08392e1") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:37:01.081239 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.080934 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fn929 for pod openshift-network-diagnostics/network-check-target-5przr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:37:01.081239 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.080986 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929 podName:4804eb98-92c2-4976-a35c-e4079e799832 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:17.080971525 +0000 UTC m=+34.274924970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn929" (UniqueName: "kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929") pod "network-check-target-5przr" (UID: "4804eb98-92c2-4976-a35c-e4079e799832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:37:01.306013 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:01.305974 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:01.306198 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:01.306023 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:01.306198 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:01.305982 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:01.306198 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.306120 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:37:01.306367 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.306229 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:37:01.306367 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:01.306290 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:37:03.309519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:03.306726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:03.314115 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:03.307054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:03.314464 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:03.314432 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:37:03.314640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:03.314476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:03.314640 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:03.314590 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:37:03.315040 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:03.315013 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:37:05.305952 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.305855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:05.305952 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.305891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:05.306477 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:05.305991 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:37:05.306477 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.305855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:05.306604 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:05.306484 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:37:05.306604 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:05.306585 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:37:05.449708 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.449664 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" event={"ID":"6ee2883a-2e3b-4546-8bb7-81d068335229","Type":"ContainerStarted","Data":"fbc0dc7a0f55a103eed832fc365df741b403df4628271651f3d95a6ffe178823"} Sep 15 20:37:05.451230 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.451189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6t5mb" event={"ID":"fa34b802-6f15-48e4-bf1b-fb01f9c2feb3","Type":"ContainerStarted","Data":"2f164175895e68685f7a1e4de0cf0e4252a9bad397a078b717d47e157cb42931"} Sep 15 20:37:05.453702 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.453417 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" event={"ID":"28a9151d-ae36-4a20-bd50-ab9db08bde8f","Type":"ContainerStarted","Data":"dc3c8fc60f08d3d020609eab24659b2ba17a78d750231e85fcd58b749ea457b8"} Sep 15 20:37:05.455471 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.455242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-895fv" event={"ID":"c22bdf6e-dac6-4791-b78d-f713a85241ea","Type":"ContainerStarted","Data":"448feaca02492a34e1c8baa668e2fc5a113de1e49967ed2d6038c42bd72a0c10"} Sep 15 20:37:05.456999 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.456972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9mm89" event={"ID":"d4e3d5cf-fff4-45f3-b820-70c6baee1155","Type":"ContainerStarted","Data":"7e4b678443b48af2f635e368b9457f5528243e86e1a3739818122f905af28524"} Sep 15 20:37:05.458620 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.458597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qblf" event={"ID":"e29a808c-8900-425b-b41b-902affb6253c","Type":"ContainerStarted","Data":"37ddb1b0d7548bc3f9be9ad1836e4197b792cf3989719978a0aa107a1bda38f6"} Sep 15 20:37:05.460146 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.460124 2579 generic.go:356] "Generic (PLEG): container finished" podID="0309b1b6-8ef8-4242-a7b0-dcd5bccb9694" containerID="c98168f53124a7364983c68207011ada11fef0d750be2fa2e57c94815ed6db63" exitCode=0 Sep 15 20:37:05.460235 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.460194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" event={"ID":"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694","Type":"ContainerDied","Data":"c98168f53124a7364983c68207011ada11fef0d750be2fa2e57c94815ed6db63"} Sep 15 20:37:05.461752 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.461730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" event={"ID":"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5","Type":"ContainerStarted","Data":"32baafaae3184c25a507d3193ceb72b8dc5df18d4b9fdf295ce5ade22d4aa93a"} Sep 15 20:37:05.464822 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.464781 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bjmhs" podStartSLOduration=2.9875931529999997 podStartE2EDuration="21.464767059s" totalImagesPullingTime="18.477173906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:05.464276856 +0000 UTC m=+22.658230312" watchObservedRunningTime="2026-09-15 20:37:05.464767059 +0000 UTC m=+22.658720547" Sep 15 20:37:05.464917 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.464896 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" podStartSLOduration=20.46488978 podStartE2EDuration="20.46488978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:36:48.425975143 +0000 UTC m=+5.619928603" watchObservedRunningTime="2026-09-15 20:37:05.46488978 +0000 UTC m=+22.658843234" Sep 15 20:37:05.503739 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.503689 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9mm89" podStartSLOduration=3.018902501 podStartE2EDuration="21.503675615s" totalImagesPullingTime="18.484773114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:05.479522911 +0000 UTC m=+22.673476360" watchObservedRunningTime="2026-09-15 20:37:05.503675615 +0000 UTC m=+22.697629069" Sep 15 20:37:05.503899 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.503793 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7qblf" podStartSLOduration=2.723637523 podStartE2EDuration="21.503789792s" totalImagesPullingTime="18.780152269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:05.503359128 +0000 UTC m=+22.697312584" watchObservedRunningTime="2026-09-15 20:37:05.503789792 +0000 UTC m=+22.697743247" Sep 15 20:37:05.536789 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.536742 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6t5mb" podStartSLOduration=3.05903552 podStartE2EDuration="21.536724979s" totalImagesPullingTime="18.477689459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:05.536315114 +0000 UTC m=+22.730268574" watchObservedRunningTime="2026-09-15 20:37:05.536724979 +0000 UTC m=+22.730678434" Sep 15 20:37:05.551298 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:05.551256 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-895fv" podStartSLOduration=3.068716132 podStartE2EDuration="21.5512388s" totalImagesPullingTime="18.482522668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:05.55115942 +0000 UTC m=+22.745112876" watchObservedRunningTime="2026-09-15 20:37:05.5512388 +0000 UTC m=+22.745192254" Sep 15 20:37:06.465880 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:06.465838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dps4j" event={"ID":"28a4cb6e-3d5d-4082-b1d8-f4aa74b2a6a0","Type":"ContainerStarted","Data":"47e211a5a8cf7808a6c27c58d7cedff6069fb6b19ec27175f32c2a35ca594bdb"} Sep 15 20:37:06.471171 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:06.471135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" event={"ID":"28a9151d-ae36-4a20-bd50-ab9db08bde8f","Type":"ContainerStarted","Data":"c21f5d6eaab086ecaf8fddefb5c81ff6af735b494efe4e3ba56f28ceadde45a5"} Sep 15 20:37:06.471323 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:06.471216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" event={"ID":"28a9151d-ae36-4a20-bd50-ab9db08bde8f","Type":"ContainerStarted","Data":"017059ebf14941a14d6bc209604d1f45a34d3831ccbf46dd60a1a4d90590fd75"} Sep 15 20:37:06.471323 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:06.471255 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" event={"ID":"28a9151d-ae36-4a20-bd50-ab9db08bde8f","Type":"ContainerStarted","Data":"e343d21fbb5e930a777d804cdae911fbbaf7a91fe02bbce5330f29e07877b841"} Sep 15 20:37:06.471323 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:06.471308 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" event={"ID":"28a9151d-ae36-4a20-bd50-ab9db08bde8f","Type":"ContainerStarted","Data":"4291a4b007b950c45826fbca16a5bf5aacf1ec183e55d604e5e3077fff5246e9"} Sep 15 20:37:06.471463 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:06.471357 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" event={"ID":"28a9151d-ae36-4a20-bd50-ab9db08bde8f","Type":"ContainerStarted","Data":"4c822fc5e5d0eb919480b5a9b623e402c7b8b1507243571b1bbeff8d4953acc3"} Sep 15 20:37:06.481283 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:06.481212 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dps4j" podStartSLOduration=3.994236681 podStartE2EDuration="22.481183635s" totalImagesPullingTime="18.486946954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:06.480802839 +0000 UTC m=+23.674756293" watchObservedRunningTime="2026-09-15 20:37:06.481183635 +0000 UTC m=+23.675137089" Sep 15 20:37:06.535724 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:06.535681 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:37:06.536915 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:06.536890 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:37:06.754645 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:06.754432 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 15 20:37:07.305509 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:07.305397 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T20:37:06.754642146Z","UUID":"eb6c05b3-4b46-40b5-aedf-a50efb016c72","Handler":null,"Name":"","Endpoint":""} Sep 15 20:37:07.305738 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:07.305554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:07.305738 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:07.305568 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:07.305738 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:07.305569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:07.305738 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:07.305674 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:37:07.305942 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:07.305784 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:37:07.305942 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:07.305885 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:37:07.307165 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:07.307145 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 15 20:37:07.307291 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:07.307179 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 15 20:37:07.475428 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:07.475395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" event={"ID":"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5","Type":"ContainerStarted","Data":"8f58079ffcea5e040c1d6189f3892773846f3fcfcf060c9236c6d27087a4c058"} Sep 15 20:37:07.475816 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:07.475803 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:37:07.476192 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:07.476172 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9mm89" Sep 15 20:37:08.480262 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:08.480220 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" event={"ID":"bba9a9cf-64f8-49fe-bbf1-f9fd858fe9c5","Type":"ContainerStarted","Data":"9036a5f4317b0251df203bc7e984ec56842328b440acfc49d565bdf2db889791"} Sep 15 20:37:08.484157 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:08.484125 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" event={"ID":"28a9151d-ae36-4a20-bd50-ab9db08bde8f","Type":"ContainerStarted","Data":"eadcdccba08cc6467fd4304c2b2f570e7329b70ab99dd3fc000239731da09bb7"} Sep 15 20:37:08.497077 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:08.497038 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dm2hl" podStartSLOduration=3.5587794 podStartE2EDuration="24.497024655s" totalImagesPullingTime="20.938245255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:08.496282285 +0000 UTC m=+25.690235740" watchObservedRunningTime="2026-09-15 20:37:08.497024655 +0000 UTC m=+25.690978109" Sep 15 20:37:09.305241 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:09.305206 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:09.305241 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:09.305230 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:09.305481 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:09.305214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:09.305481 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:09.305321 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:37:09.305481 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:09.305414 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:37:09.305660 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:09.305516 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:37:10.499672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:10.497529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" event={"ID":"28a9151d-ae36-4a20-bd50-ab9db08bde8f","Type":"ContainerStarted","Data":"1af4d3d8c1e015b26f08646eb459acac7f70b3be7f7051307f4ec248949dfce8"} Sep 15 20:37:10.499672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:10.499124 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:37:10.499672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:10.499222 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:37:10.499672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:10.499345 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:37:10.526024 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:10.524113 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:37:10.526024 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:10.524756 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" podStartSLOduration=8.006406574 podStartE2EDuration="26.524736337s" totalImagesPullingTime="18.518329763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:10.523080687 +0000 UTC m=+27.717034136" watchObservedRunningTime="2026-09-15 20:37:10.524736337 +0000 UTC m=+27.718689793" Sep 15 20:37:10.526024 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:10.525898 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:37:11.306158 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:11.305977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:11.306327 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:11.305996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:11.306327 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:11.306227 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:37:11.306327 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:11.306322 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:37:11.306533 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:11.306027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:11.306533 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:11.306397 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:37:11.501237 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:11.501192 2579 generic.go:356] "Generic (PLEG): container finished" podID="0309b1b6-8ef8-4242-a7b0-dcd5bccb9694" containerID="4872745e38a5e044b2614ba242c46b39746622ad45ef4a5948b607f601798a2c" exitCode=0 Sep 15 20:37:11.501839 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:11.501244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" event={"ID":"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694","Type":"ContainerDied","Data":"4872745e38a5e044b2614ba242c46b39746622ad45ef4a5948b607f601798a2c"} Sep 15 20:37:12.441343 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:12.441314 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h48b4"] Sep 15 20:37:12.441523 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:12.441450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:12.441645 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:12.441621 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:37:12.441975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:12.441952 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d628s"] Sep 15 20:37:12.442073 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:12.442058 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:12.442177 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:12.442158 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:37:12.442861 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:12.442781 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5przr"] Sep 15 20:37:12.442969 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:12.442871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:12.442969 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:12.442939 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:37:12.505566 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:12.505526 2579 generic.go:356] "Generic (PLEG): container finished" podID="0309b1b6-8ef8-4242-a7b0-dcd5bccb9694" containerID="086116ecac3191a0e5b9941c35c2c84800cf09d3c7177ab0259cf7cfa03ae9ac" exitCode=0 Sep 15 20:37:12.506046 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:12.505601 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" event={"ID":"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694","Type":"ContainerDied","Data":"086116ecac3191a0e5b9941c35c2c84800cf09d3c7177ab0259cf7cfa03ae9ac"} Sep 15 20:37:13.510720 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:13.510621 2579 generic.go:356] "Generic (PLEG): container finished" podID="0309b1b6-8ef8-4242-a7b0-dcd5bccb9694" containerID="7d4a6eff06e8184df16705669c4655ba1821dc84a49caecb0ac8f8b72cd4bc17" exitCode=0 Sep 15 20:37:13.510720 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:13.510681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" event={"ID":"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694","Type":"ContainerDied","Data":"7d4a6eff06e8184df16705669c4655ba1821dc84a49caecb0ac8f8b72cd4bc17"} Sep 15 20:37:14.305254 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:14.305216 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:14.305254 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:14.305247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:14.305254 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:14.305220 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:14.305487 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:14.305342 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:37:14.305487 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:14.305393 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:37:14.305487 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:14.305476 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:37:16.306143 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:16.305931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:16.306713 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:16.305965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:16.306713 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:16.306252 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h48b4" podUID="609fe877-8116-4e94-a791-ab35d08392e1" Sep 15 20:37:16.306713 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:16.306309 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:37:16.306713 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:16.305976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:16.306713 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:16.306383 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5przr" podUID="4804eb98-92c2-4976-a35c-e4079e799832" Sep 15 20:37:17.095702 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:17.095670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:17.095702 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:17.095707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn929\" (UniqueName: \"kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929\") pod \"network-check-target-5przr\" (UID: \"4804eb98-92c2-4976-a35c-e4079e799832\") " pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:17.095992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:17.095726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:17.095992 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:17.095825 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:37:17.095992 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:17.095836 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:37:17.095992 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:17.095856 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:37:17.095992 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:17.095879 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret podName:609fe877-8116-4e94-a791-ab35d08392e1 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:49.095864229 +0000 UTC m=+66.289817663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret") pod "global-pull-secret-syncer-h48b4" (UID: "609fe877-8116-4e94-a791-ab35d08392e1") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:37:17.095992 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:17.095883 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:37:17.095992 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:17.095897 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fn929 for pod openshift-network-diagnostics/network-check-target-5przr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:37:17.095992 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:17.095903 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs podName:11e01e0e-b7c2-41fb-8152-77fdd5533940 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:49.095885131 +0000 UTC m=+66.289838565 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs") pod "network-metrics-daemon-d628s" (UID: "11e01e0e-b7c2-41fb-8152-77fdd5533940") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:37:17.095992 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:17.095950 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929 podName:4804eb98-92c2-4976-a35c-e4079e799832 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:49.095936832 +0000 UTC m=+66.289890266 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn929" (UniqueName: "kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929") pod "network-check-target-5przr" (UID: "4804eb98-92c2-4976-a35c-e4079e799832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:37:18.139545 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.139519 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeReady" Sep 15 20:37:18.139952 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.139664 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 15 20:37:18.172005 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.171976 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485"] Sep 15 20:37:18.176732 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.176708 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-646ff6c755-6br8d"] Sep 15 20:37:18.176859 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.176836 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:18.179455 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.179057 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 15 20:37:18.179455 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.179297 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 15 20:37:18.179455 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.179356 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8t2g6\"" Sep 15 20:37:18.180400 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.180378 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.182159 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.182136 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 15 20:37:18.182267 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.182243 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 15 20:37:18.182337 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.182305 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 15 20:37:18.182482 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.182467 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgtm7\"" Sep 15 20:37:18.187713 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.187636 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485"] Sep 15 20:37:18.187826 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.187734 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 15 20:37:18.192928 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.192901 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wwhnn"] Sep 15 20:37:18.196981 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.196562 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vpbz5"] Sep 15 20:37:18.196981 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.196719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wwhnn" Sep 15 20:37:18.199397 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.199362 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 15 20:37:18.199610 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.199592 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 15 20:37:18.199704 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.199625 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4qdwd\"" Sep 15 20:37:18.199704 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.199629 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 15 20:37:18.200248 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.200228 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-646ff6c755-6br8d"] Sep 15 20:37:18.200468 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.200449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.201652 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.201629 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wwhnn"] Sep 15 20:37:18.202227 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.202207 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 15 20:37:18.202333 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.202230 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 15 20:37:18.202688 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.202672 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx25c\"" Sep 15 20:37:18.212203 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.212173 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpbz5"] Sep 15 20:37:18.304565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.304565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb580a9-dcac-4ef7-b922-f5d33cb41155-config-volume\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.304810 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6429a9c-1243-4d8d-a553-b971b86fe035-ca-trust-extracted\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.304810 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304634 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b799debb-e4ae-47d9-9376-66b29a395c40-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:18.304810 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccf27f42-5fe5-4492-a1b7-dfd1d26959a0-cert\") pod \"ingress-canary-wwhnn\" (UID: \"ccf27f42-5fe5-4492-a1b7-dfd1d26959a0\") " pod="openshift-ingress-canary/ingress-canary-wwhnn" Sep 15 20:37:18.304810 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-installation-pull-secrets\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.304810 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgx4x\" (UniqueName: \"kubernetes.io/projected/ccf27f42-5fe5-4492-a1b7-dfd1d26959a0-kube-api-access-rgx4x\") pod \"ingress-canary-wwhnn\" (UID: \"ccf27f42-5fe5-4492-a1b7-dfd1d26959a0\") " pod="openshift-ingress-canary/ingress-canary-wwhnn" Sep 15 20:37:18.304810 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2gbc\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-kube-api-access-c2gbc\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-image-registry-private-configuration\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-trusted-ca\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304951 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.305005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb580a9-dcac-4ef7-b922-f5d33cb41155-tmp-dir\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.305031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7wlp\" (UniqueName: \"kubernetes.io/projected/4bb580a9-dcac-4ef7-b922-f5d33cb41155-kube-api-access-p7wlp\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.304941 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.305071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-bound-sa-token\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.305106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.305107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-certificates\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.307456 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.307396 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 20:37:18.307456 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.307435 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzk8x\"" Sep 15 20:37:18.308084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.308052 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nxplx\"" Sep 15 20:37:18.308186 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.308096 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 20:37:18.308300 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.308282 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 20:37:18.309037 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.309016 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 20:37:18.405950 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.405907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-certificates\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.406157 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.405977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.406157 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.405999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb580a9-dcac-4ef7-b922-f5d33cb41155-config-volume\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.406157 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6429a9c-1243-4d8d-a553-b971b86fe035-ca-trust-extracted\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.406157 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b799debb-e4ae-47d9-9376-66b29a395c40-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:18.406367 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.406156 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:37:18.406367 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.406182 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646ff6c755-6br8d: secret "image-registry-tls" not found Sep 15 20:37:18.406367 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.406261 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls podName:c6429a9c-1243-4d8d-a553-b971b86fe035 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:18.906239695 +0000 UTC m=+36.100193141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls") pod "image-registry-646ff6c755-6br8d" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035") : secret "image-registry-tls" not found Sep 15 20:37:18.406536 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccf27f42-5fe5-4492-a1b7-dfd1d26959a0-cert\") pod \"ingress-canary-wwhnn\" (UID: \"ccf27f42-5fe5-4492-a1b7-dfd1d26959a0\") " pod="openshift-ingress-canary/ingress-canary-wwhnn" Sep 15 20:37:18.406536 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-installation-pull-secrets\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.406536 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgx4x\" (UniqueName: \"kubernetes.io/projected/ccf27f42-5fe5-4492-a1b7-dfd1d26959a0-kube-api-access-rgx4x\") pod \"ingress-canary-wwhnn\" (UID: \"ccf27f42-5fe5-4492-a1b7-dfd1d26959a0\") " pod="openshift-ingress-canary/ingress-canary-wwhnn" Sep 15 20:37:18.406536 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2gbc\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-kube-api-access-c2gbc\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.406660 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:18.406660 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-image-registry-private-configuration\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.406660 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-trusted-ca\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.406660 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-certificates\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.406660 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb580a9-dcac-4ef7-b922-f5d33cb41155-config-volume\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.406660 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.406827 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb580a9-dcac-4ef7-b922-f5d33cb41155-tmp-dir\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.406827 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b799debb-e4ae-47d9-9376-66b29a395c40-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:18.406827 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.406709 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 20:37:18.406827 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.406776 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert podName:b799debb-e4ae-47d9-9376-66b29a395c40 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:18.90675766 +0000 UTC m=+36.100711093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vg485" (UID: "b799debb-e4ae-47d9-9376-66b29a395c40") : secret "networking-console-plugin-cert" not found Sep 15 20:37:18.406827 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7wlp\" (UniqueName: \"kubernetes.io/projected/4bb580a9-dcac-4ef7-b922-f5d33cb41155-kube-api-access-p7wlp\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.406827 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.406792 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:18.406827 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.406809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-bound-sa-token\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.407029 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.406845 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls podName:4bb580a9-dcac-4ef7-b922-f5d33cb41155 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:18.906830732 +0000 UTC m=+36.100784180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls") pod "dns-default-vpbz5" (UID: "4bb580a9-dcac-4ef7-b922-f5d33cb41155") : secret "dns-default-metrics-tls" not found Sep 15 20:37:18.407376 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.407110 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb580a9-dcac-4ef7-b922-f5d33cb41155-tmp-dir\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.407376 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.407248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6429a9c-1243-4d8d-a553-b971b86fe035-ca-trust-extracted\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.407711 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.407690 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-trusted-ca\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.411938 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.411913 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-image-registry-private-configuration\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.412057 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.411938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-installation-pull-secrets\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.412057 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.412004 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccf27f42-5fe5-4492-a1b7-dfd1d26959a0-cert\") pod \"ingress-canary-wwhnn\" (UID: \"ccf27f42-5fe5-4492-a1b7-dfd1d26959a0\") " pod="openshift-ingress-canary/ingress-canary-wwhnn" Sep 15 20:37:18.418067 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.418026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2gbc\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-kube-api-access-c2gbc\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.418216 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.418172 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgx4x\" (UniqueName: \"kubernetes.io/projected/ccf27f42-5fe5-4492-a1b7-dfd1d26959a0-kube-api-access-rgx4x\") pod \"ingress-canary-wwhnn\" (UID: \"ccf27f42-5fe5-4492-a1b7-dfd1d26959a0\") " pod="openshift-ingress-canary/ingress-canary-wwhnn" Sep 15 20:37:18.418372 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.418340 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7wlp\" (UniqueName: \"kubernetes.io/projected/4bb580a9-dcac-4ef7-b922-f5d33cb41155-kube-api-access-p7wlp\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.418548 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.418520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-bound-sa-token\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.509537 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.509477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wwhnn" Sep 15 20:37:18.898102 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.897851 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wwhnn"] Sep 15 20:37:18.912105 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.912083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:18.912178 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.912169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:18.912221 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:18.912202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:18.912270 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.912231 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:37:18.912270 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.912250 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646ff6c755-6br8d: secret "image-registry-tls" not found Sep 15 20:37:18.912372 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.912299 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:18.912372 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.912305 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls podName:c6429a9c-1243-4d8d-a553-b971b86fe035 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:19.912288837 +0000 UTC m=+37.106242270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls") pod "image-registry-646ff6c755-6br8d" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035") : secret "image-registry-tls" not found Sep 15 20:37:18.912372 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.912308 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 20:37:18.912372 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.912338 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls podName:4bb580a9-dcac-4ef7-b922-f5d33cb41155 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:19.912327203 +0000 UTC m=+37.106280637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls") pod "dns-default-vpbz5" (UID: "4bb580a9-dcac-4ef7-b922-f5d33cb41155") : secret "dns-default-metrics-tls" not found Sep 15 20:37:18.912372 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:18.912363 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert podName:b799debb-e4ae-47d9-9376-66b29a395c40 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:19.912347726 +0000 UTC m=+37.106301181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vg485" (UID: "b799debb-e4ae-47d9-9376-66b29a395c40") : secret "networking-console-plugin-cert" not found Sep 15 20:37:18.920373 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:37:18.920343 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf27f42_5fe5_4492_a1b7_dfd1d26959a0.slice/crio-1fdf4190f8f2e064b3c0593d05e57ef2d9bb2f221ad92f0b45bc8c69c188e7bf WatchSource:0}: Error finding container 1fdf4190f8f2e064b3c0593d05e57ef2d9bb2f221ad92f0b45bc8c69c188e7bf: Status 404 returned error can't find the container with id 1fdf4190f8f2e064b3c0593d05e57ef2d9bb2f221ad92f0b45bc8c69c188e7bf Sep 15 20:37:19.529878 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:19.529836 2579 generic.go:356] "Generic (PLEG): container finished" podID="0309b1b6-8ef8-4242-a7b0-dcd5bccb9694" containerID="948953c8c1b1512cc68e232a374b12a679d60c988df7eaf0c1f9a53400cbebec" exitCode=0 Sep 15 20:37:19.530318 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:19.529945 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" event={"ID":"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694","Type":"ContainerDied","Data":"948953c8c1b1512cc68e232a374b12a679d60c988df7eaf0c1f9a53400cbebec"} Sep 15 20:37:19.531466 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:19.531439 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wwhnn" event={"ID":"ccf27f42-5fe5-4492-a1b7-dfd1d26959a0","Type":"ContainerStarted","Data":"1fdf4190f8f2e064b3c0593d05e57ef2d9bb2f221ad92f0b45bc8c69c188e7bf"} Sep 15 20:37:19.920174 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:19.920078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:19.920174 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:19.920172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:19.920392 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:19.920204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:19.920392 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:19.920318 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:37:19.920392 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:19.920330 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 20:37:19.920392 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:19.920342 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646ff6c755-6br8d: secret "image-registry-tls" not found Sep 15 20:37:19.920392 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:19.920330 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:19.920611 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:19.920401 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert podName:b799debb-e4ae-47d9-9376-66b29a395c40 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:21.920382822 +0000 UTC m=+39.114336276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vg485" (UID: "b799debb-e4ae-47d9-9376-66b29a395c40") : secret "networking-console-plugin-cert" not found Sep 15 20:37:19.920611 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:19.920425 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls podName:4bb580a9-dcac-4ef7-b922-f5d33cb41155 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:21.92040799 +0000 UTC m=+39.114361443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls") pod "dns-default-vpbz5" (UID: "4bb580a9-dcac-4ef7-b922-f5d33cb41155") : secret "dns-default-metrics-tls" not found Sep 15 20:37:19.920611 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:19.920474 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls podName:c6429a9c-1243-4d8d-a553-b971b86fe035 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:21.920432673 +0000 UTC m=+39.114386105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls") pod "image-registry-646ff6c755-6br8d" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035") : secret "image-registry-tls" not found Sep 15 20:37:20.538402 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:20.538343 2579 generic.go:356] "Generic (PLEG): container finished" podID="0309b1b6-8ef8-4242-a7b0-dcd5bccb9694" containerID="822437483a71f77f2999c3ef885dc924a4ac59f15e32c90f115817f5a45f637a" exitCode=0 Sep 15 20:37:20.538982 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:20.538411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" event={"ID":"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694","Type":"ContainerDied","Data":"822437483a71f77f2999c3ef885dc924a4ac59f15e32c90f115817f5a45f637a"} Sep 15 20:37:21.543831 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:21.543798 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" event={"ID":"0309b1b6-8ef8-4242-a7b0-dcd5bccb9694","Type":"ContainerStarted","Data":"096b741eb3709f90a98a9aa316352520fd19fd12d8436589c24b040871034f19"} Sep 15 20:37:21.565951 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:21.565822 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n5z2v" podStartSLOduration=7.457043046 podStartE2EDuration="37.565801589s" totalImagesPullingTime="30.108758543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:21.564955505 +0000 UTC m=+38.758908959" watchObservedRunningTime="2026-09-15 20:37:21.565801589 +0000 UTC m=+38.759755046" Sep 15 20:37:21.936376 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:21.936342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:21.936544 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:21.936397 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:21.936544 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:21.936475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:21.936544 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:21.936509 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:21.936652 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:21.936574 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls podName:4bb580a9-dcac-4ef7-b922-f5d33cb41155 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:25.936554894 +0000 UTC m=+43.130508349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls") pod "dns-default-vpbz5" (UID: "4bb580a9-dcac-4ef7-b922-f5d33cb41155") : secret "dns-default-metrics-tls" not found Sep 15 20:37:21.936652 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:21.936586 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 20:37:21.936652 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:21.936616 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:37:21.936652 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:21.936639 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646ff6c755-6br8d: secret "image-registry-tls" not found Sep 15 20:37:21.936790 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:21.936626 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert podName:b799debb-e4ae-47d9-9376-66b29a395c40 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:25.936615823 +0000 UTC m=+43.130569256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vg485" (UID: "b799debb-e4ae-47d9-9376-66b29a395c40") : secret "networking-console-plugin-cert" not found Sep 15 20:37:21.936790 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:21.936713 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls podName:c6429a9c-1243-4d8d-a553-b971b86fe035 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:25.93668875 +0000 UTC m=+43.130642195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls") pod "image-registry-646ff6c755-6br8d" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035") : secret "image-registry-tls" not found Sep 15 20:37:22.547247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:22.547202 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wwhnn" event={"ID":"ccf27f42-5fe5-4492-a1b7-dfd1d26959a0","Type":"ContainerStarted","Data":"8320371f6883be652cc149d339c509fe909c083f7d1c55bc6bfee542226de496"} Sep 15 20:37:22.561611 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:22.561563 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wwhnn" podStartSLOduration=1.629142061 podStartE2EDuration="4.561546812s" totalImagesPullingTime="2.932404751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:22.560958298 +0000 UTC m=+39.754911756" watchObservedRunningTime="2026-09-15 20:37:22.561546812 +0000 UTC m=+39.755500266" Sep 15 20:37:25.967781 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:25.967737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:25.967781 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:25.967783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:25.968198 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:25.967817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:25.968198 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:25.967904 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:37:25.968198 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:25.967906 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 20:37:25.968198 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:25.967915 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646ff6c755-6br8d: secret "image-registry-tls" not found Sep 15 20:37:25.968198 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:25.967904 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:25.968198 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:25.967969 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert podName:b799debb-e4ae-47d9-9376-66b29a395c40 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:33.967953046 +0000 UTC m=+51.161906480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vg485" (UID: "b799debb-e4ae-47d9-9376-66b29a395c40") : secret "networking-console-plugin-cert" not found Sep 15 20:37:25.968198 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:25.967985 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls podName:4bb580a9-dcac-4ef7-b922-f5d33cb41155 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:33.967974951 +0000 UTC m=+51.161928385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls") pod "dns-default-vpbz5" (UID: "4bb580a9-dcac-4ef7-b922-f5d33cb41155") : secret "dns-default-metrics-tls" not found Sep 15 20:37:25.968198 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:25.967996 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls podName:c6429a9c-1243-4d8d-a553-b971b86fe035 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:33.967990957 +0000 UTC m=+51.161944389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls") pod "image-registry-646ff6c755-6br8d" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035") : secret "image-registry-tls" not found Sep 15 20:37:34.030122 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:34.030071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:34.030772 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:34.030149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:34.030772 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:34.030176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:34.030772 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:34.030219 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:37:34.030772 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:34.030242 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646ff6c755-6br8d: secret "image-registry-tls" not found Sep 15 20:37:34.030772 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:34.030275 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:34.030772 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:34.030276 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 20:37:34.030772 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:34.030301 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls podName:c6429a9c-1243-4d8d-a553-b971b86fe035 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:50.030285224 +0000 UTC m=+67.224238673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls") pod "image-registry-646ff6c755-6br8d" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035") : secret "image-registry-tls" not found Sep 15 20:37:34.030772 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:34.030320 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls podName:4bb580a9-dcac-4ef7-b922-f5d33cb41155 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:50.030307912 +0000 UTC m=+67.224261344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls") pod "dns-default-vpbz5" (UID: "4bb580a9-dcac-4ef7-b922-f5d33cb41155") : secret "dns-default-metrics-tls" not found Sep 15 20:37:34.030772 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:34.030332 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert podName:b799debb-e4ae-47d9-9376-66b29a395c40 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:50.030326483 +0000 UTC m=+67.224279916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vg485" (UID: "b799debb-e4ae-47d9-9376-66b29a395c40") : secret "networking-console-plugin-cert" not found Sep 15 20:37:42.516368 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:42.516326 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xmfnc" Sep 15 20:37:49.142312 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.142268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:37:49.142312 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.142312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn929\" (UniqueName: \"kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929\") pod \"network-check-target-5przr\" (UID: \"4804eb98-92c2-4976-a35c-e4079e799832\") " pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:49.142876 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.142331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:49.144746 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.144728 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 20:37:49.144809 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.144729 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 20:37:49.144809 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.144729 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 20:37:49.153433 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:49.153403 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 15 20:37:49.153591 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:49.153515 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs podName:11e01e0e-b7c2-41fb-8152-77fdd5533940 nodeName:}" failed. No retries permitted until 2026-09-15 20:38:53.153472604 +0000 UTC m=+130.347426039 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs") pod "network-metrics-daemon-d628s" (UID: "11e01e0e-b7c2-41fb-8152-77fdd5533940") : secret "metrics-daemon-secret" not found Sep 15 20:37:49.155688 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.155672 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 20:37:49.156817 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.156800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/609fe877-8116-4e94-a791-ab35d08392e1-original-pull-secret\") pod \"global-pull-secret-syncer-h48b4\" (UID: \"609fe877-8116-4e94-a791-ab35d08392e1\") " pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:49.165863 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.165834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn929\" (UniqueName: \"kubernetes.io/projected/4804eb98-92c2-4976-a35c-e4079e799832-kube-api-access-fn929\") pod \"network-check-target-5przr\" (UID: \"4804eb98-92c2-4976-a35c-e4079e799832\") " pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:49.224698 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.224663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h48b4" Sep 15 20:37:49.232609 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.232572 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzk8x\"" Sep 15 20:37:49.240576 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.240548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:49.529073 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.529044 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h48b4"] Sep 15 20:37:49.532095 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:37:49.532071 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609fe877_8116_4e94_a791_ab35d08392e1.slice/crio-3cf2086b535ebc68e59e324d3b6fbe9543c88ffc9140025ae4525c09d73a05e7 WatchSource:0}: Error finding container 3cf2086b535ebc68e59e324d3b6fbe9543c88ffc9140025ae4525c09d73a05e7: Status 404 returned error can't find the container with id 3cf2086b535ebc68e59e324d3b6fbe9543c88ffc9140025ae4525c09d73a05e7 Sep 15 20:37:49.554101 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.554075 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5przr"] Sep 15 20:37:49.556986 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:37:49.556956 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4804eb98_92c2_4976_a35c_e4079e799832.slice/crio-dfd06b027e58ffa5f29921fab4eb520b29124b76d2d3ec38041be17230e399ec WatchSource:0}: Error finding container dfd06b027e58ffa5f29921fab4eb520b29124b76d2d3ec38041be17230e399ec: Status 404 returned error can't find the container with id dfd06b027e58ffa5f29921fab4eb520b29124b76d2d3ec38041be17230e399ec Sep 15 20:37:49.608270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.608236 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5przr" event={"ID":"4804eb98-92c2-4976-a35c-e4079e799832","Type":"ContainerStarted","Data":"dfd06b027e58ffa5f29921fab4eb520b29124b76d2d3ec38041be17230e399ec"} Sep 15 20:37:49.609329 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:49.609307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h48b4" event={"ID":"609fe877-8116-4e94-a791-ab35d08392e1","Type":"ContainerStarted","Data":"3cf2086b535ebc68e59e324d3b6fbe9543c88ffc9140025ae4525c09d73a05e7"} Sep 15 20:37:50.047616 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:50.047577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:37:50.047814 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:50.047631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:37:50.047814 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:50.047677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:37:50.047814 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:50.047735 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 20:37:50.047814 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:50.047797 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:50.047814 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:50.047801 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:37:50.048026 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:50.047823 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646ff6c755-6br8d: secret "image-registry-tls" not found Sep 15 20:37:50.048026 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:50.047800 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert podName:b799debb-e4ae-47d9-9376-66b29a395c40 nodeName:}" failed. No retries permitted until 2026-09-15 20:38:22.047785497 +0000 UTC m=+99.241738930 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vg485" (UID: "b799debb-e4ae-47d9-9376-66b29a395c40") : secret "networking-console-plugin-cert" not found Sep 15 20:37:50.048026 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:50.047878 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls podName:4bb580a9-dcac-4ef7-b922-f5d33cb41155 nodeName:}" failed. No retries permitted until 2026-09-15 20:38:22.047862664 +0000 UTC m=+99.241816111 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls") pod "dns-default-vpbz5" (UID: "4bb580a9-dcac-4ef7-b922-f5d33cb41155") : secret "dns-default-metrics-tls" not found Sep 15 20:37:50.048026 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:37:50.047894 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls podName:c6429a9c-1243-4d8d-a553-b971b86fe035 nodeName:}" failed. No retries permitted until 2026-09-15 20:38:22.047883792 +0000 UTC m=+99.241837230 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls") pod "image-registry-646ff6c755-6br8d" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035") : secret "image-registry-tls" not found Sep 15 20:37:54.623480 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:54.623435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5przr" event={"ID":"4804eb98-92c2-4976-a35c-e4079e799832","Type":"ContainerStarted","Data":"619a9bceda46e55d0eb9917d84f131fa32351befaf200633513922268c75ed2d"} Sep 15 20:37:54.623952 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:54.623559 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:37:54.625009 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:54.624985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h48b4" event={"ID":"609fe877-8116-4e94-a791-ab35d08392e1","Type":"ContainerStarted","Data":"48bbcb18f4d5d0e19718e64f0acf02cb3d12bfab9d35feb63db378c6499de8c4"} Sep 15 20:37:54.638087 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:54.638047 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5przr" podStartSLOduration=65.758660204 podStartE2EDuration="1m10.638033288s" totalImagesPullingTime="4.879373084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:54.637461689 +0000 UTC m=+71.831415145" watchObservedRunningTime="2026-09-15 20:37:54.638033288 +0000 UTC m=+71.831986737" Sep 15 20:37:54.651172 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:37:54.651125 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h48b4" podStartSLOduration=65.742664004 podStartE2EDuration="1m10.651111007s" totalImagesPullingTime="4.908447003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:54.651000479 +0000 UTC m=+71.844953934" watchObservedRunningTime="2026-09-15 20:37:54.651111007 +0000 UTC m=+71.845064461" Sep 15 20:38:01.633168 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.633130 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544"] Sep 15 20:38:01.637243 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.637224 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:01.639311 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.639290 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 15 20:38:01.640257 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.640236 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 15 20:38:01.640364 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.640245 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 15 20:38:01.640364 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.640305 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 15 20:38:01.643942 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.643911 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544"] Sep 15 20:38:01.736644 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.736603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtwbn\" (UniqueName: \"kubernetes.io/projected/9bf8f574-98db-42b5-abd2-fbc137dcc73b-kube-api-access-dtwbn\") pod \"klusterlet-addon-workmgr-8489d75d8b-2t544\" (UID: \"9bf8f574-98db-42b5-abd2-fbc137dcc73b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:01.736831 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.736681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bf8f574-98db-42b5-abd2-fbc137dcc73b-tmp\") pod \"klusterlet-addon-workmgr-8489d75d8b-2t544\" (UID: \"9bf8f574-98db-42b5-abd2-fbc137dcc73b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:01.736831 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.736734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9bf8f574-98db-42b5-abd2-fbc137dcc73b-klusterlet-config\") pod \"klusterlet-addon-workmgr-8489d75d8b-2t544\" (UID: \"9bf8f574-98db-42b5-abd2-fbc137dcc73b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:01.837533 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.837479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9bf8f574-98db-42b5-abd2-fbc137dcc73b-klusterlet-config\") pod \"klusterlet-addon-workmgr-8489d75d8b-2t544\" (UID: \"9bf8f574-98db-42b5-abd2-fbc137dcc73b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:01.837533 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.837539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtwbn\" (UniqueName: \"kubernetes.io/projected/9bf8f574-98db-42b5-abd2-fbc137dcc73b-kube-api-access-dtwbn\") pod \"klusterlet-addon-workmgr-8489d75d8b-2t544\" (UID: \"9bf8f574-98db-42b5-abd2-fbc137dcc73b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:01.837780 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.837587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bf8f574-98db-42b5-abd2-fbc137dcc73b-tmp\") pod \"klusterlet-addon-workmgr-8489d75d8b-2t544\" (UID: \"9bf8f574-98db-42b5-abd2-fbc137dcc73b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:01.837902 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.837883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bf8f574-98db-42b5-abd2-fbc137dcc73b-tmp\") pod \"klusterlet-addon-workmgr-8489d75d8b-2t544\" (UID: \"9bf8f574-98db-42b5-abd2-fbc137dcc73b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:01.840154 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.840128 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9bf8f574-98db-42b5-abd2-fbc137dcc73b-klusterlet-config\") pod \"klusterlet-addon-workmgr-8489d75d8b-2t544\" (UID: \"9bf8f574-98db-42b5-abd2-fbc137dcc73b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:01.845136 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.845104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtwbn\" (UniqueName: \"kubernetes.io/projected/9bf8f574-98db-42b5-abd2-fbc137dcc73b-kube-api-access-dtwbn\") pod \"klusterlet-addon-workmgr-8489d75d8b-2t544\" (UID: \"9bf8f574-98db-42b5-abd2-fbc137dcc73b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:01.946958 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:01.946871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:02.211684 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:02.211611 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544"] Sep 15 20:38:02.214682 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:38:02.214656 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf8f574_98db_42b5_abd2_fbc137dcc73b.slice/crio-cb18e5a36824032cf9d2294a5107b1869f7ff2251a6a687e9ac1d8b15b612656 WatchSource:0}: Error finding container cb18e5a36824032cf9d2294a5107b1869f7ff2251a6a687e9ac1d8b15b612656: Status 404 returned error can't find the container with id cb18e5a36824032cf9d2294a5107b1869f7ff2251a6a687e9ac1d8b15b612656 Sep 15 20:38:02.644907 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:02.644873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" event={"ID":"9bf8f574-98db-42b5-abd2-fbc137dcc73b","Type":"ContainerStarted","Data":"cb18e5a36824032cf9d2294a5107b1869f7ff2251a6a687e9ac1d8b15b612656"} Sep 15 20:38:06.656690 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:06.656650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" event={"ID":"9bf8f574-98db-42b5-abd2-fbc137dcc73b","Type":"ContainerStarted","Data":"815647ef567c83437e80d72932d14eed0a18d85e623820511692977622061442"} Sep 15 20:38:06.657158 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:06.656857 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:06.658683 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:06.658661 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:38:06.673624 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:06.673568 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" podStartSLOduration=1.856894966 podStartE2EDuration="5.673554582s" totalImagesPullingTime="3.816659616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:38:06.673024362 +0000 UTC m=+83.866977816" watchObservedRunningTime="2026-09-15 20:38:06.673554582 +0000 UTC m=+83.867508037" Sep 15 20:38:22.087450 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:22.087403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:38:22.087450 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:22.087453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:38:22.087968 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:22.087516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:38:22.087968 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:38:22.087602 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:38:22.087968 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:38:22.087604 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 20:38:22.087968 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:38:22.087666 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls podName:4bb580a9-dcac-4ef7-b922-f5d33cb41155 nodeName:}" failed. No retries permitted until 2026-09-15 20:39:26.087650455 +0000 UTC m=+163.281603909 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls") pod "dns-default-vpbz5" (UID: "4bb580a9-dcac-4ef7-b922-f5d33cb41155") : secret "dns-default-metrics-tls" not found Sep 15 20:38:22.087968 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:38:22.087699 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert podName:b799debb-e4ae-47d9-9376-66b29a395c40 nodeName:}" failed. No retries permitted until 2026-09-15 20:39:26.087680333 +0000 UTC m=+163.281633780 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vg485" (UID: "b799debb-e4ae-47d9-9376-66b29a395c40") : secret "networking-console-plugin-cert" not found Sep 15 20:38:22.087968 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:38:22.087606 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:38:22.087968 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:38:22.087718 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646ff6c755-6br8d: secret "image-registry-tls" not found Sep 15 20:38:22.087968 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:38:22.087758 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls podName:c6429a9c-1243-4d8d-a553-b971b86fe035 nodeName:}" failed. No retries permitted until 2026-09-15 20:39:26.087748138 +0000 UTC m=+163.281701584 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls") pod "image-registry-646ff6c755-6br8d" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035") : secret "image-registry-tls" not found Sep 15 20:38:25.630128 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:25.630096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5przr" Sep 15 20:38:53.209441 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:38:53.209406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:38:53.209916 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:38:53.209563 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 15 20:38:53.209916 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:38:53.209629 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs podName:11e01e0e-b7c2-41fb-8152-77fdd5533940 nodeName:}" failed. No retries permitted until 2026-09-15 20:40:55.209612857 +0000 UTC m=+252.403566290 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs") pod "network-metrics-daemon-d628s" (UID: "11e01e0e-b7c2-41fb-8152-77fdd5533940") : secret "metrics-daemon-secret" not found Sep 15 20:39:21.191441 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:21.191396 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" podUID="b799debb-e4ae-47d9-9376-66b29a395c40" Sep 15 20:39:21.198689 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:21.198659 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" podUID="c6429a9c-1243-4d8d-a553-b971b86fe035" Sep 15 20:39:21.216971 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:21.216941 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vpbz5" podUID="4bb580a9-dcac-4ef7-b922-f5d33cb41155" Sep 15 20:39:21.317393 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:21.317357 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:39:21.834802 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:39:21.834767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpbz5" Sep 15 20:39:21.834997 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:39:21.834767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:39:21.834997 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:39:21.834767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:39:26.149556 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:39:26.149515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:39:26.150010 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:39:26.149582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:39:26.150010 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:39:26.149605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:39:26.150010 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:26.149691 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:39:26.150010 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:26.149713 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646ff6c755-6br8d: secret "image-registry-tls" not found Sep 15 20:39:26.150010 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:26.149731 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 20:39:26.150010 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:26.149699 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:39:26.150010 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:26.149780 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls podName:c6429a9c-1243-4d8d-a553-b971b86fe035 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:28.14976283 +0000 UTC m=+285.343716262 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls") pod "image-registry-646ff6c755-6br8d" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035") : secret "image-registry-tls" not found Sep 15 20:39:26.150010 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:26.149797 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert podName:b799debb-e4ae-47d9-9376-66b29a395c40 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:28.149788381 +0000 UTC m=+285.343741816 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vg485" (UID: "b799debb-e4ae-47d9-9376-66b29a395c40") : secret "networking-console-plugin-cert" not found Sep 15 20:39:26.150010 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:39:26.149816 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls podName:4bb580a9-dcac-4ef7-b922-f5d33cb41155 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:28.149805187 +0000 UTC m=+285.343758620 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls") pod "dns-default-vpbz5" (UID: "4bb580a9-dcac-4ef7-b922-f5d33cb41155") : secret "dns-default-metrics-tls" not found Sep 15 20:39:36.306033 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:39:36.305929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:40:06.658134 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:40:06.658087 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" podUID="9bf8f574-98db-42b5-abd2-fbc137dcc73b" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused" Sep 15 20:40:06.942120 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:40:06.942040 2579 generic.go:356] "Generic (PLEG): container finished" podID="9bf8f574-98db-42b5-abd2-fbc137dcc73b" containerID="815647ef567c83437e80d72932d14eed0a18d85e623820511692977622061442" exitCode=1 Sep 15 20:40:06.942249 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:40:06.942116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" event={"ID":"9bf8f574-98db-42b5-abd2-fbc137dcc73b","Type":"ContainerDied","Data":"815647ef567c83437e80d72932d14eed0a18d85e623820511692977622061442"} Sep 15 20:40:06.942444 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:40:06.942422 2579 scope.go:117] "RemoveContainer" containerID="815647ef567c83437e80d72932d14eed0a18d85e623820511692977622061442" Sep 15 20:40:07.945886 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:40:07.945856 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" event={"ID":"9bf8f574-98db-42b5-abd2-fbc137dcc73b","Type":"ContainerStarted","Data":"dc315b42d982dc604bfc22096f4e0f1374024ccd536b6f3f2ce9cdef12d846dd"} Sep 15 20:40:07.946250 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:40:07.946147 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:40:07.946770 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:40:07.946750 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8489d75d8b-2t544" Sep 15 20:40:55.292435 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:40:55.292379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:40:55.293047 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:40:55.292579 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 15 20:40:55.293047 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:40:55.292663 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs podName:11e01e0e-b7c2-41fb-8152-77fdd5533940 nodeName:}" failed. No retries permitted until 2026-09-15 20:42:57.292642394 +0000 UTC m=+374.486595846 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs") pod "network-metrics-daemon-d628s" (UID: "11e01e0e-b7c2-41fb-8152-77fdd5533940") : secret "metrics-daemon-secret" not found Sep 15 20:41:00.786762 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.786729 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z"] Sep 15 20:41:00.789839 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.789814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:00.790206 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.790002 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr"] Sep 15 20:41:00.791951 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.791927 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 15 20:41:00.792100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.792085 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 15 20:41:00.792882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.792865 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch"] Sep 15 20:41:00.792979 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.792963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:00.794398 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.794374 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mhcm7\"" Sep 15 20:41:00.794518 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.794404 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 15 20:41:00.794518 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.794380 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 15 20:41:00.795481 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.795466 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:00.798834 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.798811 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 15 20:41:00.798834 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.798829 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:41:00.798986 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.798928 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 15 20:41:00.798986 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.798928 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:41:00.799089 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.798993 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6vmgj\"" Sep 15 20:41:00.799089 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.799008 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 15 20:41:00.799089 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.799041 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 15 20:41:00.799223 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.799111 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mn2zd\"" Sep 15 20:41:00.799223 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.799165 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 15 20:41:00.799223 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.799180 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 15 20:41:00.802668 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.802649 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z"] Sep 15 20:41:00.806553 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.806491 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr"] Sep 15 20:41:00.807577 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.807558 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch"] Sep 15 20:41:00.832575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.832553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7a7aacf-c2e6-40ba-bd51-c943817eda80-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:00.832713 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.832582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7cm\" (UniqueName: \"kubernetes.io/projected/f7a7aacf-c2e6-40ba-bd51-c943817eda80-kube-api-access-2b7cm\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:00.832713 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.832611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:00.832713 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.832626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64f3e5db-c556-4406-827d-b1401e9ce442-serving-cert\") pod \"service-ca-operator-cf7c776b7-m8jgr\" (UID: \"64f3e5db-c556-4406-827d-b1401e9ce442\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:00.832713 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.832695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64f3e5db-c556-4406-827d-b1401e9ce442-config\") pod \"service-ca-operator-cf7c776b7-m8jgr\" (UID: \"64f3e5db-c556-4406-827d-b1401e9ce442\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:00.832872 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.832722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/167ff948-f63b-42d8-b67e-882ddafb2a0d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wfzch\" (UID: \"167ff948-f63b-42d8-b67e-882ddafb2a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:00.832872 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.832743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbnj4\" (UniqueName: \"kubernetes.io/projected/167ff948-f63b-42d8-b67e-882ddafb2a0d-kube-api-access-bbnj4\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wfzch\" (UID: \"167ff948-f63b-42d8-b67e-882ddafb2a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:00.832872 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.832802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbdmk\" (UniqueName: \"kubernetes.io/projected/64f3e5db-c556-4406-827d-b1401e9ce442-kube-api-access-dbdmk\") pod \"service-ca-operator-cf7c776b7-m8jgr\" (UID: \"64f3e5db-c556-4406-827d-b1401e9ce442\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:00.832872 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.832857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/167ff948-f63b-42d8-b67e-882ddafb2a0d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wfzch\" (UID: \"167ff948-f63b-42d8-b67e-882ddafb2a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:00.933621 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.933589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/167ff948-f63b-42d8-b67e-882ddafb2a0d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wfzch\" (UID: \"167ff948-f63b-42d8-b67e-882ddafb2a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:00.933791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.933673 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7a7aacf-c2e6-40ba-bd51-c943817eda80-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:00.933791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.933693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7cm\" (UniqueName: \"kubernetes.io/projected/f7a7aacf-c2e6-40ba-bd51-c943817eda80-kube-api-access-2b7cm\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:00.933791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.933721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:00.933791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.933737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64f3e5db-c556-4406-827d-b1401e9ce442-serving-cert\") pod \"service-ca-operator-cf7c776b7-m8jgr\" (UID: \"64f3e5db-c556-4406-827d-b1401e9ce442\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:00.933791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.933754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64f3e5db-c556-4406-827d-b1401e9ce442-config\") pod \"service-ca-operator-cf7c776b7-m8jgr\" (UID: \"64f3e5db-c556-4406-827d-b1401e9ce442\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:00.933791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.933770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/167ff948-f63b-42d8-b67e-882ddafb2a0d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wfzch\" (UID: \"167ff948-f63b-42d8-b67e-882ddafb2a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:00.933791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.933788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbnj4\" (UniqueName: \"kubernetes.io/projected/167ff948-f63b-42d8-b67e-882ddafb2a0d-kube-api-access-bbnj4\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wfzch\" (UID: \"167ff948-f63b-42d8-b67e-882ddafb2a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:00.934174 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.933816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbdmk\" (UniqueName: \"kubernetes.io/projected/64f3e5db-c556-4406-827d-b1401e9ce442-kube-api-access-dbdmk\") pod \"service-ca-operator-cf7c776b7-m8jgr\" (UID: \"64f3e5db-c556-4406-827d-b1401e9ce442\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:00.934174 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:00.933834 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:00.934174 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:00.933917 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls podName:f7a7aacf-c2e6-40ba-bd51-c943817eda80 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:01.433895714 +0000 UTC m=+258.627849160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xr58z" (UID: "f7a7aacf-c2e6-40ba-bd51-c943817eda80") : secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:00.934434 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.934409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/167ff948-f63b-42d8-b67e-882ddafb2a0d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wfzch\" (UID: \"167ff948-f63b-42d8-b67e-882ddafb2a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:00.934538 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.934490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7a7aacf-c2e6-40ba-bd51-c943817eda80-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:00.934648 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.934632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64f3e5db-c556-4406-827d-b1401e9ce442-config\") pod \"service-ca-operator-cf7c776b7-m8jgr\" (UID: \"64f3e5db-c556-4406-827d-b1401e9ce442\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:00.936619 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.936599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64f3e5db-c556-4406-827d-b1401e9ce442-serving-cert\") pod \"service-ca-operator-cf7c776b7-m8jgr\" (UID: \"64f3e5db-c556-4406-827d-b1401e9ce442\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:00.936722 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.936618 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/167ff948-f63b-42d8-b67e-882ddafb2a0d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wfzch\" (UID: \"167ff948-f63b-42d8-b67e-882ddafb2a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:00.941915 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.941884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbnj4\" (UniqueName: \"kubernetes.io/projected/167ff948-f63b-42d8-b67e-882ddafb2a0d-kube-api-access-bbnj4\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wfzch\" (UID: \"167ff948-f63b-42d8-b67e-882ddafb2a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:00.942095 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.942071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7cm\" (UniqueName: \"kubernetes.io/projected/f7a7aacf-c2e6-40ba-bd51-c943817eda80-kube-api-access-2b7cm\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:00.942652 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:00.942635 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbdmk\" (UniqueName: \"kubernetes.io/projected/64f3e5db-c556-4406-827d-b1401e9ce442-kube-api-access-dbdmk\") pod \"service-ca-operator-cf7c776b7-m8jgr\" (UID: \"64f3e5db-c556-4406-827d-b1401e9ce442\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:01.107320 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:01.107230 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" Sep 15 20:41:01.112409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:01.112388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" Sep 15 20:41:01.414855 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:01.414760 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch"] Sep 15 20:41:01.415697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:01.415677 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr"] Sep 15 20:41:01.419630 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:01.419517 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f3e5db_c556_4406_827d_b1401e9ce442.slice/crio-bb2493a40294fa4310f8704d05bdbb20778ba180551a5c57e60b963b62459558 WatchSource:0}: Error finding container bb2493a40294fa4310f8704d05bdbb20778ba180551a5c57e60b963b62459558: Status 404 returned error can't find the container with id bb2493a40294fa4310f8704d05bdbb20778ba180551a5c57e60b963b62459558 Sep 15 20:41:01.420252 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:01.420225 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167ff948_f63b_42d8_b67e_882ddafb2a0d.slice/crio-37a5475e90e29b4248915f952a11fa3db248eea45f4e2c15ec76acb7d3d2125e WatchSource:0}: Error finding container 37a5475e90e29b4248915f952a11fa3db248eea45f4e2c15ec76acb7d3d2125e: Status 404 returned error can't find the container with id 37a5475e90e29b4248915f952a11fa3db248eea45f4e2c15ec76acb7d3d2125e Sep 15 20:41:01.436992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:01.436969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:01.437108 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:01.437081 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:01.437148 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:01.437126 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls podName:f7a7aacf-c2e6-40ba-bd51-c943817eda80 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:02.437112884 +0000 UTC m=+259.631066317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xr58z" (UID: "f7a7aacf-c2e6-40ba-bd51-c943817eda80") : secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:02.077662 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:02.077631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" event={"ID":"64f3e5db-c556-4406-827d-b1401e9ce442","Type":"ContainerStarted","Data":"bb2493a40294fa4310f8704d05bdbb20778ba180551a5c57e60b963b62459558"} Sep 15 20:41:02.078660 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:02.078638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" event={"ID":"167ff948-f63b-42d8-b67e-882ddafb2a0d","Type":"ContainerStarted","Data":"37a5475e90e29b4248915f952a11fa3db248eea45f4e2c15ec76acb7d3d2125e"} Sep 15 20:41:02.445627 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:02.445545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:02.445803 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:02.445718 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:02.445803 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:02.445796 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls podName:f7a7aacf-c2e6-40ba-bd51-c943817eda80 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:04.445775233 +0000 UTC m=+261.639728669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xr58z" (UID: "f7a7aacf-c2e6-40ba-bd51-c943817eda80") : secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:04.460367 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:04.460274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:04.460748 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:04.460442 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:04.460748 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:04.460546 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls podName:f7a7aacf-c2e6-40ba-bd51-c943817eda80 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.460524557 +0000 UTC m=+265.654477996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xr58z" (UID: "f7a7aacf-c2e6-40ba-bd51-c943817eda80") : secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:05.089293 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.089254 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" event={"ID":"64f3e5db-c556-4406-827d-b1401e9ce442","Type":"ContainerStarted","Data":"aa2352d0407e11329e1c135668c8a5127b4ad0bec20027804bc5f6eec9cf534a"} Sep 15 20:41:05.090719 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.090693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" event={"ID":"167ff948-f63b-42d8-b67e-882ddafb2a0d","Type":"ContainerStarted","Data":"482f0a8d005ddf96bfb86463b60040a726f06285a56583eff92d224e6a98868d"} Sep 15 20:41:05.105671 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.105621 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" podStartSLOduration=2.267710392 podStartE2EDuration="5.105600991s" totalImagesPullingTime="2.837890599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:05.104295941 +0000 UTC m=+262.298249396" watchObservedRunningTime="2026-09-15 20:41:05.105600991 +0000 UTC m=+262.299554448" Sep 15 20:41:05.130992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.130939 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" podStartSLOduration=2.295940443 podStartE2EDuration="5.130919988s" totalImagesPullingTime="2.834979545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:05.129817201 +0000 UTC m=+262.323770657" watchObservedRunningTime="2026-09-15 20:41:05.130919988 +0000 UTC m=+262.324873445" Sep 15 20:41:05.183149 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.183116 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4246q"] Sep 15 20:41:05.186888 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.186866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.192009 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.191986 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 15 20:41:05.192130 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.192095 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 15 20:41:05.192595 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.192579 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 15 20:41:05.192766 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.192631 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d898r\"" Sep 15 20:41:05.192766 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.192658 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 15 20:41:05.200201 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.200179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4246q"] Sep 15 20:41:05.267362 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.267326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.267533 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.267395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shpff\" (UniqueName: \"kubernetes.io/projected/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-api-access-shpff\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.267533 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.267470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3e7a137-86c0-4bbf-a96e-718e87443852-crio-socket\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.267637 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.267545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.267637 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.267572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e7a137-86c0-4bbf-a96e-718e87443852-data-volume\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.368532 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.368427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3e7a137-86c0-4bbf-a96e-718e87443852-crio-socket\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.368532 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.368482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.368747 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:05.368562 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 20:41:05.368747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.368590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3e7a137-86c0-4bbf-a96e-718e87443852-crio-socket\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.368747 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:05.368629 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:05.868615329 +0000 UTC m=+263.062568763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : configmap "kube-rbac-proxy" not found Sep 15 20:41:05.368747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.368620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e7a137-86c0-4bbf-a96e-718e87443852-data-volume\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.368747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.368712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.368947 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.368752 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shpff\" (UniqueName: \"kubernetes.io/projected/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-api-access-shpff\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.368947 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:05.368809 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:05.368947 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:05.368877 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:05.868863657 +0000 UTC m=+263.062817091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:05.368947 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.368926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e7a137-86c0-4bbf-a96e-718e87443852-data-volume\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.376584 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.376560 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shpff\" (UniqueName: \"kubernetes.io/projected/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-api-access-shpff\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.860118 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.860078 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64"] Sep 15 20:41:05.863248 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.863231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64" Sep 15 20:41:05.865531 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.865481 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 15 20:41:05.866139 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.866113 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bm4dh\"" Sep 15 20:41:05.866240 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.866152 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 15 20:41:05.872547 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.872524 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64"] Sep 15 20:41:05.873697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.873672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.873789 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:05.873774 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 20:41:05.873789 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.873779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:05.873889 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:05.873822 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:06.873808569 +0000 UTC m=+264.067762002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : configmap "kube-rbac-proxy" not found Sep 15 20:41:05.873889 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:05.873884 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:05.873989 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:05.873937 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:06.87391618 +0000 UTC m=+264.067869621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:05.974793 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:05.974758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthmk\" (UniqueName: \"kubernetes.io/projected/adb5bd85-37c1-4626-a581-e952a9409205-kube-api-access-mthmk\") pod \"migrator-8b4d697c5-kzx64\" (UID: \"adb5bd85-37c1-4626-a581-e952a9409205\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64" Sep 15 20:41:06.075700 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:06.075667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthmk\" (UniqueName: \"kubernetes.io/projected/adb5bd85-37c1-4626-a581-e952a9409205-kube-api-access-mthmk\") pod \"migrator-8b4d697c5-kzx64\" (UID: \"adb5bd85-37c1-4626-a581-e952a9409205\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64" Sep 15 20:41:06.084212 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:06.084179 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthmk\" (UniqueName: \"kubernetes.io/projected/adb5bd85-37c1-4626-a581-e952a9409205-kube-api-access-mthmk\") pod \"migrator-8b4d697c5-kzx64\" (UID: \"adb5bd85-37c1-4626-a581-e952a9409205\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64" Sep 15 20:41:06.173071 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:06.172970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64" Sep 15 20:41:06.438948 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:06.438868 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64"] Sep 15 20:41:06.442904 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:06.442876 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb5bd85_37c1_4626_a581_e952a9409205.slice/crio-1734c88fc4821827ceafe97fa848327345b1758a429b1f2657a66325a780c8cc WatchSource:0}: Error finding container 1734c88fc4821827ceafe97fa848327345b1758a429b1f2657a66325a780c8cc: Status 404 returned error can't find the container with id 1734c88fc4821827ceafe97fa848327345b1758a429b1f2657a66325a780c8cc Sep 15 20:41:06.882956 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:06.882919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:06.883370 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:06.883019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:06.883370 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:06.883039 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 20:41:06.883370 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:06.883106 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.883085584 +0000 UTC m=+266.077039034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : configmap "kube-rbac-proxy" not found Sep 15 20:41:06.883370 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:06.883159 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:06.883370 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:06.883211 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.88319829 +0000 UTC m=+266.077151756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:07.097842 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:07.097802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64" event={"ID":"adb5bd85-37c1-4626-a581-e952a9409205","Type":"ContainerStarted","Data":"1734c88fc4821827ceafe97fa848327345b1758a429b1f2657a66325a780c8cc"} Sep 15 20:41:07.133304 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:07.133223 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6t5mb_fa34b802-6f15-48e4-bf1b-fb01f9c2feb3/dns-node-resolver/0.log" Sep 15 20:41:07.932369 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:07.932345 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-895fv_c22bdf6e-dac6-4791-b78d-f713a85241ea/node-ca/0.log" Sep 15 20:41:08.103440 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:08.103407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64" event={"ID":"adb5bd85-37c1-4626-a581-e952a9409205","Type":"ContainerStarted","Data":"b639fdc2cedfe1a26c164b5ded3c1ff4392c2b57301ffacbf88b068f0dda8edb"} Sep 15 20:41:08.103639 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:08.103459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64" event={"ID":"adb5bd85-37c1-4626-a581-e952a9409205","Type":"ContainerStarted","Data":"3bc1806225a880dab81b67234d86d954e2421e17794694a7507d003525f57e8a"} Sep 15 20:41:08.119273 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:08.119226 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kzx64" podStartSLOduration=1.8749726880000002 podStartE2EDuration="3.119206193s" totalImagesPullingTime="1.244233505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:08.118568468 +0000 UTC m=+265.312521924" watchObservedRunningTime="2026-09-15 20:41:08.119206193 +0000 UTC m=+265.313159649" Sep 15 20:41:08.495358 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:08.495264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:08.495551 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:08.495408 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:08.495551 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:08.495472 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls podName:f7a7aacf-c2e6-40ba-bd51-c943817eda80 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:16.495457761 +0000 UTC m=+273.689411199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xr58z" (UID: "f7a7aacf-c2e6-40ba-bd51-c943817eda80") : secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:08.898985 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:08.898952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:08.899147 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:08.899023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:08.899147 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:08.899080 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 20:41:08.899147 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:08.899134 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:08.899147 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:08.899142 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:12.899127539 +0000 UTC m=+270.093080972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : configmap "kube-rbac-proxy" not found Sep 15 20:41:08.899310 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:08.899173 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:12.899162742 +0000 UTC m=+270.093116175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:09.130973 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:09.130942 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wwhnn_ccf27f42-5fe5-4492-a1b7-dfd1d26959a0/serve-healthcheck-canary/0.log" Sep 15 20:41:12.930427 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:12.930392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:12.930847 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:12.930460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:12.930847 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:12.930537 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 20:41:12.930847 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:12.930566 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:12.930847 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:12.930589 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:20.930574914 +0000 UTC m=+278.124528347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : configmap "kube-rbac-proxy" not found Sep 15 20:41:12.930847 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:12.930617 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:20.930603812 +0000 UTC m=+278.124557245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:16.562387 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:16.562346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:16.562771 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:16.562454 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:16.562771 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:16.562535 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls podName:f7a7aacf-c2e6-40ba-bd51-c943817eda80 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:32.562516388 +0000 UTC m=+289.756469839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xr58z" (UID: "f7a7aacf-c2e6-40ba-bd51-c943817eda80") : secret "cluster-monitoring-operator-tls" not found Sep 15 20:41:20.999612 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:20.999571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:20.999992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:20.999640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:20.999992 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:20.999734 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:20.999992 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:20.999800 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls podName:a3e7a137-86c0-4bbf-a96e-718e87443852 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:36.999784984 +0000 UTC m=+294.193738417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4246q" (UID: "a3e7a137-86c0-4bbf-a96e-718e87443852") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:21.000098 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:21.000065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e7a137-86c0-4bbf-a96e-718e87443852-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:24.835929 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:24.835878 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vpbz5" podUID="4bb580a9-dcac-4ef7-b922-f5d33cb41155" Sep 15 20:41:24.835929 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:24.835880 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" podUID="c6429a9c-1243-4d8d-a553-b971b86fe035" Sep 15 20:41:24.836339 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:24.835963 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" podUID="b799debb-e4ae-47d9-9376-66b29a395c40" Sep 15 20:41:25.151332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:25.151261 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpbz5" Sep 15 20:41:25.151454 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:25.151261 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:41:25.151454 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:25.151261 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:41:28.156601 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.156561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:41:28.156601 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.156601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:41:28.157180 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.156667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:41:28.159261 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.159235 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"image-registry-646ff6c755-6br8d\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:41:28.159261 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.159269 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b799debb-e4ae-47d9-9376-66b29a395c40-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vg485\" (UID: \"b799debb-e4ae-47d9-9376-66b29a395c40\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:41:28.159435 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.159358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bb580a9-dcac-4ef7-b922-f5d33cb41155-metrics-tls\") pod \"dns-default-vpbz5\" (UID: \"4bb580a9-dcac-4ef7-b922-f5d33cb41155\") " pod="openshift-dns/dns-default-vpbz5" Sep 15 20:41:28.455854 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.455760 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8t2g6\"" Sep 15 20:41:28.455854 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.455760 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgtm7\"" Sep 15 20:41:28.455854 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.455827 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx25c\"" Sep 15 20:41:28.462850 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.462819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" Sep 15 20:41:28.462850 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.462839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpbz5" Sep 15 20:41:28.462996 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.462930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:41:28.745069 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.745032 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485"] Sep 15 20:41:28.747402 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.746289 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpbz5"] Sep 15 20:41:28.749160 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:28.749126 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb580a9_dcac_4ef7_b922_f5d33cb41155.slice/crio-5440bdd901c418f533819779fd6816285c21948ec581fc798ffb2ddef9f09ac0 WatchSource:0}: Error finding container 5440bdd901c418f533819779fd6816285c21948ec581fc798ffb2ddef9f09ac0: Status 404 returned error can't find the container with id 5440bdd901c418f533819779fd6816285c21948ec581fc798ffb2ddef9f09ac0 Sep 15 20:41:28.749381 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:28.749361 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb799debb_e4ae_47d9_9376_66b29a395c40.slice/crio-3d16c2d68c6b91c84325713155a259e4f8af28366d0d1e7184dcddc910674f4d WatchSource:0}: Error finding container 3d16c2d68c6b91c84325713155a259e4f8af28366d0d1e7184dcddc910674f4d: Status 404 returned error can't find the container with id 3d16c2d68c6b91c84325713155a259e4f8af28366d0d1e7184dcddc910674f4d Sep 15 20:41:28.754135 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:28.754114 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-646ff6c755-6br8d"] Sep 15 20:41:28.756594 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:28.756574 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6429a9c_1243_4d8d_a553_b971b86fe035.slice/crio-17e22081a9825b7f5482b322027ef681b6763a8615aedc3aa0d6a3666c43c1d7 WatchSource:0}: Error finding container 17e22081a9825b7f5482b322027ef681b6763a8615aedc3aa0d6a3666c43c1d7: Status 404 returned error can't find the container with id 17e22081a9825b7f5482b322027ef681b6763a8615aedc3aa0d6a3666c43c1d7 Sep 15 20:41:29.165611 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:29.165459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" event={"ID":"c6429a9c-1243-4d8d-a553-b971b86fe035","Type":"ContainerStarted","Data":"592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2"} Sep 15 20:41:29.166070 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:29.165627 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:41:29.166070 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:29.165670 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" event={"ID":"c6429a9c-1243-4d8d-a553-b971b86fe035","Type":"ContainerStarted","Data":"17e22081a9825b7f5482b322027ef681b6763a8615aedc3aa0d6a3666c43c1d7"} Sep 15 20:41:29.167139 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:29.167109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" event={"ID":"b799debb-e4ae-47d9-9376-66b29a395c40","Type":"ContainerStarted","Data":"3d16c2d68c6b91c84325713155a259e4f8af28366d0d1e7184dcddc910674f4d"} Sep 15 20:41:29.168606 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:29.168581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpbz5" event={"ID":"4bb580a9-dcac-4ef7-b922-f5d33cb41155","Type":"ContainerStarted","Data":"5440bdd901c418f533819779fd6816285c21948ec581fc798ffb2ddef9f09ac0"} Sep 15 20:41:29.188651 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:29.188608 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" podStartSLOduration=285.188595373 podStartE2EDuration="4m45.188595373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:29.188152954 +0000 UTC m=+286.382106409" watchObservedRunningTime="2026-09-15 20:41:29.188595373 +0000 UTC m=+286.382548857" Sep 15 20:41:30.173592 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:30.173556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" event={"ID":"b799debb-e4ae-47d9-9376-66b29a395c40","Type":"ContainerStarted","Data":"0ec11d1d54b97d3bb334f4611f061d18bf318a3f4567b40da0ee329465ff2217"} Sep 15 20:41:30.189127 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:30.189066 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vg485" podStartSLOduration=275.906391399 podStartE2EDuration="4m37.189048547s" totalImagesPullingTime="1.282657148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:30.188256061 +0000 UTC m=+287.382209518" watchObservedRunningTime="2026-09-15 20:41:30.189048547 +0000 UTC m=+287.383002001" Sep 15 20:41:31.178224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:31.178192 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpbz5" event={"ID":"4bb580a9-dcac-4ef7-b922-f5d33cb41155","Type":"ContainerStarted","Data":"d83ddfa08b4098237f2c53d6d4685df1ccada547a185c43ebce6a844b186dbd3"} Sep 15 20:41:31.178644 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:31.178243 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpbz5" event={"ID":"4bb580a9-dcac-4ef7-b922-f5d33cb41155","Type":"ContainerStarted","Data":"b277840479f452d07a4d8b05d924d349f4ebe5f1342683cdcdd68634a4ac4b1e"} Sep 15 20:41:31.178644 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:31.178382 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vpbz5" Sep 15 20:41:31.194621 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:31.194567 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vpbz5" podStartSLOduration=251.269478224 podStartE2EDuration="4m13.194549663s" totalImagesPullingTime="1.925071439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:31.193630049 +0000 UTC m=+288.387583504" watchObservedRunningTime="2026-09-15 20:41:31.194549663 +0000 UTC m=+288.388503118" Sep 15 20:41:32.590868 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:32.590832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:32.593446 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:32.593421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7a7aacf-c2e6-40ba-bd51-c943817eda80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xr58z\" (UID: \"f7a7aacf-c2e6-40ba-bd51-c943817eda80\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:32.600296 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:32.600278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" Sep 15 20:41:32.866470 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:32.866398 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z"] Sep 15 20:41:32.869540 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:32.869512 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a7aacf_c2e6_40ba_bd51_c943817eda80.slice/crio-b1d14a4dbe0606ddcc3e1e4ec28ba8b4cec81a639fb86294157be0a582c93c16 WatchSource:0}: Error finding container b1d14a4dbe0606ddcc3e1e4ec28ba8b4cec81a639fb86294157be0a582c93c16: Status 404 returned error can't find the container with id b1d14a4dbe0606ddcc3e1e4ec28ba8b4cec81a639fb86294157be0a582c93c16 Sep 15 20:41:33.186999 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:33.186925 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" event={"ID":"f7a7aacf-c2e6-40ba-bd51-c943817eda80","Type":"ContainerStarted","Data":"b1d14a4dbe0606ddcc3e1e4ec28ba8b4cec81a639fb86294157be0a582c93c16"} Sep 15 20:41:34.943594 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:34.943556 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq"] Sep 15 20:41:34.946748 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:34.946726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq" Sep 15 20:41:34.948839 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:34.948813 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8qnjh\"" Sep 15 20:41:34.948942 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:34.948851 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 15 20:41:34.956199 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:34.956175 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq"] Sep 15 20:41:35.110607 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:35.110567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/649b7cd2-d289-4a44-928a-1bb1968a17fc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-wzjnq\" (UID: \"649b7cd2-d289-4a44-928a-1bb1968a17fc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq" Sep 15 20:41:35.195928 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:35.195847 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" event={"ID":"f7a7aacf-c2e6-40ba-bd51-c943817eda80","Type":"ContainerStarted","Data":"97ac01a96525c9683be23814d9bde23de7fc22f393575591725690408def818a"} Sep 15 20:41:35.210992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:35.210938 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xr58z" podStartSLOduration=33.618375817 podStartE2EDuration="35.210917148s" totalImagesPullingTime="1.592541331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:35.210342455 +0000 UTC m=+292.404295911" watchObservedRunningTime="2026-09-15 20:41:35.210917148 +0000 UTC m=+292.404870604" Sep 15 20:41:35.210992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:35.210983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/649b7cd2-d289-4a44-928a-1bb1968a17fc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-wzjnq\" (UID: \"649b7cd2-d289-4a44-928a-1bb1968a17fc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq" Sep 15 20:41:35.213756 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:35.213728 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/649b7cd2-d289-4a44-928a-1bb1968a17fc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-wzjnq\" (UID: \"649b7cd2-d289-4a44-928a-1bb1968a17fc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq" Sep 15 20:41:35.255274 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:35.255240 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq" Sep 15 20:41:35.525161 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:35.524800 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq"] Sep 15 20:41:35.527340 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:35.527303 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649b7cd2_d289_4a44_928a_1bb1968a17fc.slice/crio-d83027c1209f1495342165734f1ce07e7bec22141bb0652beba691544f6a05ca WatchSource:0}: Error finding container d83027c1209f1495342165734f1ce07e7bec22141bb0652beba691544f6a05ca: Status 404 returned error can't find the container with id d83027c1209f1495342165734f1ce07e7bec22141bb0652beba691544f6a05ca Sep 15 20:41:36.201536 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:36.201480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq" event={"ID":"649b7cd2-d289-4a44-928a-1bb1968a17fc","Type":"ContainerStarted","Data":"d83027c1209f1495342165734f1ce07e7bec22141bb0652beba691544f6a05ca"} Sep 15 20:41:37.026092 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:37.026049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:37.028734 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:37.028710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e7a137-86c0-4bbf-a96e-718e87443852-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4246q\" (UID: \"a3e7a137-86c0-4bbf-a96e-718e87443852\") " pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:37.209975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:37.209936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq" event={"ID":"649b7cd2-d289-4a44-928a-1bb1968a17fc","Type":"ContainerStarted","Data":"b2727c23f7492dbbee13d02892f6c43a5a849a8330d8bfb1051c1a09be58aec4"} Sep 15 20:41:37.210411 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:37.210022 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq" Sep 15 20:41:37.214831 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:37.214801 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq" Sep 15 20:41:37.224942 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:37.224891 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wzjnq" podStartSLOduration=2.108288509 podStartE2EDuration="3.224874536s" totalImagesPullingTime="1.116586027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:37.224561035 +0000 UTC m=+294.418514489" watchObservedRunningTime="2026-09-15 20:41:37.224874536 +0000 UTC m=+294.418827992" Sep 15 20:41:37.297202 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:37.297125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4246q" Sep 15 20:41:37.565769 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:37.565694 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4246q"] Sep 15 20:41:37.568539 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:37.568516 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e7a137_86c0_4bbf_a96e_718e87443852.slice/crio-86b1bf7ba0a56c9cf6e5ae74a8a4b9170ae5cc43b002f06a1c97142257dde678 WatchSource:0}: Error finding container 86b1bf7ba0a56c9cf6e5ae74a8a4b9170ae5cc43b002f06a1c97142257dde678: Status 404 returned error can't find the container with id 86b1bf7ba0a56c9cf6e5ae74a8a4b9170ae5cc43b002f06a1c97142257dde678 Sep 15 20:41:38.214848 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:38.214814 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4246q" event={"ID":"a3e7a137-86c0-4bbf-a96e-718e87443852","Type":"ContainerStarted","Data":"ca55fe6325d7f0bb4751cfb47416f51bace1db55b33e0d508bbf3816200b6e69"} Sep 15 20:41:38.215224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:38.214865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4246q" event={"ID":"a3e7a137-86c0-4bbf-a96e-718e87443852","Type":"ContainerStarted","Data":"86b1bf7ba0a56c9cf6e5ae74a8a4b9170ae5cc43b002f06a1c97142257dde678"} Sep 15 20:41:39.219600 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:39.219564 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4246q" event={"ID":"a3e7a137-86c0-4bbf-a96e-718e87443852","Type":"ContainerStarted","Data":"7bc9eedd6f8e1491d2688bbbca8736b91a828ef475ebcd2467b0e04ba69d95c1"} Sep 15 20:41:39.306406 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:39.306364 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-d628s" podUID="11e01e0e-b7c2-41fb-8152-77fdd5533940" Sep 15 20:41:41.184043 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:41.184015 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vpbz5" Sep 15 20:41:41.229443 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:41.229411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4246q" event={"ID":"a3e7a137-86c0-4bbf-a96e-718e87443852","Type":"ContainerStarted","Data":"8818f48544150e2698cae79b3a6311aff1c9e9073452df51df240a334dbbad2a"} Sep 15 20:41:41.245963 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:41.245921 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4246q" podStartSLOduration=33.496738552 podStartE2EDuration="36.245904939s" totalImagesPullingTime="2.749166387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:41:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:41.245038693 +0000 UTC m=+298.438992149" watchObservedRunningTime="2026-09-15 20:41:41.245904939 +0000 UTC m=+298.439858394" Sep 15 20:41:43.242533 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:43.242507 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 15 20:41:44.333967 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.333938 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jsgz8"] Sep 15 20:41:44.337300 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.337284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.339637 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.339602 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 15 20:41:44.339768 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.339676 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 15 20:41:44.339917 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.339899 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qq8fj\"" Sep 15 20:41:44.340423 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.340405 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 15 20:41:44.340423 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.340420 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 15 20:41:44.382560 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.382522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1266b01d-c256-4e1f-b4b1-c613c4168b83-root\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.382560 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.382566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.382830 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.382605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-accelerators-collector-config\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.382830 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.382648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-wtmp\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.382830 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.382663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1266b01d-c256-4e1f-b4b1-c613c4168b83-metrics-client-ca\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.382830 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.382683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-textfile\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.382830 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.382816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbfz\" (UniqueName: \"kubernetes.io/projected/1266b01d-c256-4e1f-b4b1-c613c4168b83-kube-api-access-dhbfz\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.383034 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.382857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1266b01d-c256-4e1f-b4b1-c613c4168b83-sys\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.383034 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.382890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-tls\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.483709 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.483667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-accelerators-collector-config\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.483905 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.483733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-wtmp\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.483905 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.483758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1266b01d-c256-4e1f-b4b1-c613c4168b83-metrics-client-ca\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.483905 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.483781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-textfile\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.483905 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.483837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbfz\" (UniqueName: \"kubernetes.io/projected/1266b01d-c256-4e1f-b4b1-c613c4168b83-kube-api-access-dhbfz\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.484105 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.483938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-wtmp\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.484105 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.483964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1266b01d-c256-4e1f-b4b1-c613c4168b83-sys\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.484105 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.484002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-tls\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.484105 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.484072 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1266b01d-c256-4e1f-b4b1-c613c4168b83-sys\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.484105 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:44.484086 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 15 20:41:44.484303 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.484125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1266b01d-c256-4e1f-b4b1-c613c4168b83-root\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.484303 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.484101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1266b01d-c256-4e1f-b4b1-c613c4168b83-root\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.484303 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.484132 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-textfile\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.484303 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:41:44.484161 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-tls podName:1266b01d-c256-4e1f-b4b1-c613c4168b83 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:44.984140252 +0000 UTC m=+302.178093702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-tls") pod "node-exporter-jsgz8" (UID: "1266b01d-c256-4e1f-b4b1-c613c4168b83") : secret "node-exporter-tls" not found Sep 15 20:41:44.484303 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.484196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.484459 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.484314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1266b01d-c256-4e1f-b4b1-c613c4168b83-metrics-client-ca\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.484459 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.484356 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-accelerators-collector-config\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.486631 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.486592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.492975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.492953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbfz\" (UniqueName: \"kubernetes.io/projected/1266b01d-c256-4e1f-b4b1-c613c4168b83-kube-api-access-dhbfz\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.987979 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.987944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-tls\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:44.990374 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:44.990351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1266b01d-c256-4e1f-b4b1-c613c4168b83-node-exporter-tls\") pod \"node-exporter-jsgz8\" (UID: \"1266b01d-c256-4e1f-b4b1-c613c4168b83\") " pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:45.246230 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:45.246156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jsgz8" Sep 15 20:41:45.258008 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:45.257979 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1266b01d_c256_4e1f_b4b1_c613c4168b83.slice/crio-8fd0b83e45b2eb42d1812df6a9d1ac2eb5451bdbf57654d3f211e372ec02de07 WatchSource:0}: Error finding container 8fd0b83e45b2eb42d1812df6a9d1ac2eb5451bdbf57654d3f211e372ec02de07: Status 404 returned error can't find the container with id 8fd0b83e45b2eb42d1812df6a9d1ac2eb5451bdbf57654d3f211e372ec02de07 Sep 15 20:41:45.260106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:45.260084 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 20:41:46.246817 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:46.246775 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsgz8" event={"ID":"1266b01d-c256-4e1f-b4b1-c613c4168b83","Type":"ContainerStarted","Data":"8fd0b83e45b2eb42d1812df6a9d1ac2eb5451bdbf57654d3f211e372ec02de07"} Sep 15 20:41:47.252221 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:47.252176 2579 generic.go:356] "Generic (PLEG): container finished" podID="1266b01d-c256-4e1f-b4b1-c613c4168b83" containerID="6e79956d592a59cc3b29b8cc0bf2a522807581dd858b2541c7d2fc07b8f0fa0a" exitCode=0 Sep 15 20:41:47.252648 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:47.252268 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsgz8" event={"ID":"1266b01d-c256-4e1f-b4b1-c613c4168b83","Type":"ContainerDied","Data":"6e79956d592a59cc3b29b8cc0bf2a522807581dd858b2541c7d2fc07b8f0fa0a"} Sep 15 20:41:48.257621 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.257587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsgz8" event={"ID":"1266b01d-c256-4e1f-b4b1-c613c4168b83","Type":"ContainerStarted","Data":"d53e503a7c37632e9e53836c1f3985cb8973fa0859f8e4951f6176f864d8675e"} Sep 15 20:41:48.257975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.257637 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsgz8" event={"ID":"1266b01d-c256-4e1f-b4b1-c613c4168b83","Type":"ContainerStarted","Data":"933228f6698149635de12317500268343050a23fdb45e40e04883a8cb1536cff"} Sep 15 20:41:48.277962 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.277905 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jsgz8" podStartSLOduration=3.223092646 podStartE2EDuration="4.277890188s" totalImagesPullingTime="986.993659ms" totalInitContainerRuntime="67.803883ms" isStatefulPod=true podCreationTimestamp="2026-09-15 20:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:48.276850108 +0000 UTC m=+305.470803563" watchObservedRunningTime="2026-09-15 20:41:48.277890188 +0000 UTC m=+305.471843641" Sep 15 20:41:48.467449 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.467415 2579 patch_prober.go:28] interesting pod/image-registry-646ff6c755-6br8d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 15 20:41:48.467634 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.467470 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" podUID="c6429a9c-1243-4d8d-a553-b971b86fe035" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 20:41:48.771963 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.771929 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g"] Sep 15 20:41:48.775044 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.775029 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.778024 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.777999 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 15 20:41:48.778234 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.778206 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-25mphur603idp\"" Sep 15 20:41:48.778367 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.778347 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8rzjn\"" Sep 15 20:41:48.778453 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.778437 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 15 20:41:48.778592 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.778576 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 15 20:41:48.778655 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.778616 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 15 20:41:48.785447 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.785428 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g"] Sep 15 20:41:48.819668 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.819635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0c6e358f-b24a-4510-9a92-fb290819e97a-audit-log\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.819802 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.819683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6e358f-b24a-4510-9a92-fb290819e97a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.819802 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.819741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0c6e358f-b24a-4510-9a92-fb290819e97a-secret-metrics-server-client-certs\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.819802 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.819772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0c6e358f-b24a-4510-9a92-fb290819e97a-metrics-server-audit-profiles\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.819920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.819824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0c6e358f-b24a-4510-9a92-fb290819e97a-secret-metrics-server-tls\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.819920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.819860 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnz5\" (UniqueName: \"kubernetes.io/projected/0c6e358f-b24a-4510-9a92-fb290819e97a-kube-api-access-4fnz5\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.819920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.819895 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0c6e358f-b24a-4510-9a92-fb290819e97a-client-ca-bundle\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.920968 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.920922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6e358f-b24a-4510-9a92-fb290819e97a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.921175 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.920980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0c6e358f-b24a-4510-9a92-fb290819e97a-secret-metrics-server-client-certs\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.921175 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.921004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0c6e358f-b24a-4510-9a92-fb290819e97a-metrics-server-audit-profiles\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.921175 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.921035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0c6e358f-b24a-4510-9a92-fb290819e97a-secret-metrics-server-tls\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.921175 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.921056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnz5\" (UniqueName: \"kubernetes.io/projected/0c6e358f-b24a-4510-9a92-fb290819e97a-kube-api-access-4fnz5\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.921175 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.921089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0c6e358f-b24a-4510-9a92-fb290819e97a-client-ca-bundle\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.921175 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.921108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0c6e358f-b24a-4510-9a92-fb290819e97a-audit-log\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.921586 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.921558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0c6e358f-b24a-4510-9a92-fb290819e97a-audit-log\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.921877 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.921828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6e358f-b24a-4510-9a92-fb290819e97a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.922215 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.922194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0c6e358f-b24a-4510-9a92-fb290819e97a-metrics-server-audit-profiles\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.923674 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.923649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0c6e358f-b24a-4510-9a92-fb290819e97a-secret-metrics-server-tls\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.923761 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.923736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0c6e358f-b24a-4510-9a92-fb290819e97a-secret-metrics-server-client-certs\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.923918 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.923900 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0c6e358f-b24a-4510-9a92-fb290819e97a-client-ca-bundle\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:48.928864 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:48.928839 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnz5\" (UniqueName: \"kubernetes.io/projected/0c6e358f-b24a-4510-9a92-fb290819e97a-kube-api-access-4fnz5\") pod \"metrics-server-6d9d59fdf9-4kj7g\" (UID: \"0c6e358f-b24a-4510-9a92-fb290819e97a\") " pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:49.084238 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.084163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:41:49.101845 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.101818 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m"] Sep 15 20:41:49.105096 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.105077 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" Sep 15 20:41:49.108458 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.108436 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 15 20:41:49.108614 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.108595 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x497j\"" Sep 15 20:41:49.115080 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.115060 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m"] Sep 15 20:41:49.224120 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.224083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmh7v\" (UniqueName: \"kubernetes.io/projected/ba8fe8ea-5355-4a6a-8551-140f398a10a3-kube-api-access-cmh7v\") pod \"monitoring-plugin-5d85778b9d-cmn6m\" (UID: \"ba8fe8ea-5355-4a6a-8551-140f398a10a3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" Sep 15 20:41:49.224283 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.224132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba8fe8ea-5355-4a6a-8551-140f398a10a3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cmn6m\" (UID: \"ba8fe8ea-5355-4a6a-8551-140f398a10a3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" Sep 15 20:41:49.325072 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.325037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmh7v\" (UniqueName: \"kubernetes.io/projected/ba8fe8ea-5355-4a6a-8551-140f398a10a3-kube-api-access-cmh7v\") pod \"monitoring-plugin-5d85778b9d-cmn6m\" (UID: \"ba8fe8ea-5355-4a6a-8551-140f398a10a3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" Sep 15 20:41:49.325557 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.325228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba8fe8ea-5355-4a6a-8551-140f398a10a3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cmn6m\" (UID: \"ba8fe8ea-5355-4a6a-8551-140f398a10a3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" Sep 15 20:41:49.327915 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.327887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba8fe8ea-5355-4a6a-8551-140f398a10a3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cmn6m\" (UID: \"ba8fe8ea-5355-4a6a-8551-140f398a10a3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" Sep 15 20:41:49.332638 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.332613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmh7v\" (UniqueName: \"kubernetes.io/projected/ba8fe8ea-5355-4a6a-8551-140f398a10a3-kube-api-access-cmh7v\") pod \"monitoring-plugin-5d85778b9d-cmn6m\" (UID: \"ba8fe8ea-5355-4a6a-8551-140f398a10a3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" Sep 15 20:41:49.354628 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.354571 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g"] Sep 15 20:41:49.357458 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:49.357435 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6e358f_b24a_4510_9a92_fb290819e97a.slice/crio-4f6d4a2260d2f7513917f26f6edb85a120c2b8ba58e8b98012628a60ab2cee34 WatchSource:0}: Error finding container 4f6d4a2260d2f7513917f26f6edb85a120c2b8ba58e8b98012628a60ab2cee34: Status 404 returned error can't find the container with id 4f6d4a2260d2f7513917f26f6edb85a120c2b8ba58e8b98012628a60ab2cee34 Sep 15 20:41:49.414974 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.414938 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" Sep 15 20:41:49.684799 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:49.684721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m"] Sep 15 20:41:49.687805 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:49.687777 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8fe8ea_5355_4a6a_8551_140f398a10a3.slice/crio-d6258182b6c1d71542eb43d15c57f4bfd4e676bd38e5d824befc80b11498a4b6 WatchSource:0}: Error finding container d6258182b6c1d71542eb43d15c57f4bfd4e676bd38e5d824befc80b11498a4b6: Status 404 returned error can't find the container with id d6258182b6c1d71542eb43d15c57f4bfd4e676bd38e5d824befc80b11498a4b6 Sep 15 20:41:50.178552 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.178523 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:41:50.267056 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.267015 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" event={"ID":"ba8fe8ea-5355-4a6a-8551-140f398a10a3","Type":"ContainerStarted","Data":"d6258182b6c1d71542eb43d15c57f4bfd4e676bd38e5d824befc80b11498a4b6"} Sep 15 20:41:50.268762 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.268736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" event={"ID":"0c6e358f-b24a-4510-9a92-fb290819e97a","Type":"ContainerStarted","Data":"4f6d4a2260d2f7513917f26f6edb85a120c2b8ba58e8b98012628a60ab2cee34"} Sep 15 20:41:50.477969 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.477873 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 15 20:41:50.482017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.481984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.484385 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.484359 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 15 20:41:50.484907 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.484667 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 15 20:41:50.485080 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.484998 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 15 20:41:50.485080 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.485017 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 15 20:41:50.485207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.485154 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7hbfm\"" Sep 15 20:41:50.485207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.485166 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 15 20:41:50.485304 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.485247 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 15 20:41:50.486148 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.485727 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 15 20:41:50.486148 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.485945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 15 20:41:50.486148 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.485993 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 15 20:41:50.486148 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.486065 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 15 20:41:50.487770 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.487400 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 15 20:41:50.487770 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.487645 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 15 20:41:50.487948 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.487857 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-epqppoagjaouo\"" Sep 15 20:41:50.488007 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.487983 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 15 20:41:50.489038 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.489017 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 15 20:41:50.504011 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.503979 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 15 20:41:50.536390 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.536574 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.536574 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.536574 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.536574 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.536574 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.536574 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8qv\" (UniqueName: \"kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-kube-api-access-7v8qv\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.536574 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.536574 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536551 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-config\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-config-out\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.537008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.536888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-web-config\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.638398 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.638349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.638398 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.638399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.638687 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.638608 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.638687 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.638659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.638808 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.638691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-config\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.638808 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.638716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.638808 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.638748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-config-out\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-web-config\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639436 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639584 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639664 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8qv\" (UniqueName: \"kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-kube-api-access-7v8qv\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639728 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.639933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.639842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.640888 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.640525 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.641013 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.640957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.644343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.644691 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-config\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.644941 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.645044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.645163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.646857 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-web-config\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.647469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.648044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.648157 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.648657 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.648778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.648729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.649792 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.649769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8qv\" (UniqueName: \"kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-kube-api-access-7v8qv\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.649900 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.649809 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-config-out\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.650949 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.650924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:50.795653 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:50.795597 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:41:51.197838 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:51.197769 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 15 20:41:51.282659 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:41:51.282628 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2539cf_5689_42bb_996a_189f35a62da8.slice/crio-c617a4839cfdc8eca0938c4ae1157d826e24da75700c1997fe961ff3b6c895e6 WatchSource:0}: Error finding container c617a4839cfdc8eca0938c4ae1157d826e24da75700c1997fe961ff3b6c895e6: Status 404 returned error can't find the container with id c617a4839cfdc8eca0938c4ae1157d826e24da75700c1997fe961ff3b6c895e6 Sep 15 20:41:51.305951 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:51.305926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:41:51.595002 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:51.594971 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-646ff6c755-6br8d"] Sep 15 20:41:52.277875 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:52.277836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" event={"ID":"ba8fe8ea-5355-4a6a-8551-140f398a10a3","Type":"ContainerStarted","Data":"aed44055932011864baf75f1ac8e3b09fb9f493165d238157b7c6c60995bc45e"} Sep 15 20:41:52.278018 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:52.277943 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" Sep 15 20:41:52.279561 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:52.279534 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" event={"ID":"0c6e358f-b24a-4510-9a92-fb290819e97a","Type":"ContainerStarted","Data":"66f6a2fd35bbfa1871203e84223d8c105d982e4c9b2fdfa9d4b0aac2ec96cd3a"} Sep 15 20:41:52.281392 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:52.281365 2579 generic.go:356] "Generic (PLEG): container finished" podID="4e2539cf-5689-42bb-996a-189f35a62da8" containerID="a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590" exitCode=0 Sep 15 20:41:52.281573 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:52.281428 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerDied","Data":"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590"} Sep 15 20:41:52.281573 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:52.281511 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerStarted","Data":"c617a4839cfdc8eca0938c4ae1157d826e24da75700c1997fe961ff3b6c895e6"} Sep 15 20:41:52.283743 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:52.283718 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" Sep 15 20:41:52.293642 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:52.293448 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cmn6m" podStartSLOduration=1.6527372009999999 podStartE2EDuration="3.293421809s" totalImagesPullingTime="1.640684608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:52.293302522 +0000 UTC m=+309.487255981" watchObservedRunningTime="2026-09-15 20:41:52.293421809 +0000 UTC m=+309.487375265" Sep 15 20:41:52.321085 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:52.321006 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" podStartSLOduration=2.3915692809999998 podStartE2EDuration="4.320985577s" totalImagesPullingTime="1.929416296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:52.320782495 +0000 UTC m=+309.514735950" watchObservedRunningTime="2026-09-15 20:41:52.320985577 +0000 UTC m=+309.514939037" Sep 15 20:41:56.301082 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:56.301046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerStarted","Data":"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9"} Sep 15 20:41:56.301476 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:56.301111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerStarted","Data":"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08"} Sep 15 20:41:58.312256 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:58.312219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerStarted","Data":"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e"} Sep 15 20:41:58.312787 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:58.312291 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerStarted","Data":"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70"} Sep 15 20:41:58.312787 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:58.312331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerStarted","Data":"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979"} Sep 15 20:41:58.312787 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:58.312371 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerStarted","Data":"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2"} Sep 15 20:41:58.340434 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:41:58.340347 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.309045465 podStartE2EDuration="8.34029611s" totalImagesPullingTime="6.014504828s" totalInitContainerRuntime="16.745817ms" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:58.338147139 +0000 UTC m=+315.532100593" watchObservedRunningTime="2026-09-15 20:41:58.34029611 +0000 UTC m=+315.534249568" Sep 15 20:42:00.796148 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:00.796114 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:42:09.085171 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:09.085140 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:42:09.085665 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:09.085198 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:42:16.614803 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.614762 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" podUID="c6429a9c-1243-4d8d-a553-b971b86fe035" containerName="registry" containerID="cri-o://592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2" gracePeriod=30 Sep 15 20:42:16.866763 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.866704 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:42:16.973032 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.973001 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-certificates\") pod \"c6429a9c-1243-4d8d-a553-b971b86fe035\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " Sep 15 20:42:16.973198 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.973084 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2gbc\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-kube-api-access-c2gbc\") pod \"c6429a9c-1243-4d8d-a553-b971b86fe035\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " Sep 15 20:42:16.973198 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.973128 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-bound-sa-token\") pod \"c6429a9c-1243-4d8d-a553-b971b86fe035\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " Sep 15 20:42:16.973198 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.973180 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-installation-pull-secrets\") pod \"c6429a9c-1243-4d8d-a553-b971b86fe035\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " Sep 15 20:42:16.973370 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.973216 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-trusted-ca\") pod \"c6429a9c-1243-4d8d-a553-b971b86fe035\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " Sep 15 20:42:16.973370 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.973261 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6429a9c-1243-4d8d-a553-b971b86fe035-ca-trust-extracted\") pod \"c6429a9c-1243-4d8d-a553-b971b86fe035\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " Sep 15 20:42:16.973370 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.973298 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-image-registry-private-configuration\") pod \"c6429a9c-1243-4d8d-a553-b971b86fe035\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " Sep 15 20:42:16.973370 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.973329 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") pod \"c6429a9c-1243-4d8d-a553-b971b86fe035\" (UID: \"c6429a9c-1243-4d8d-a553-b971b86fe035\") " Sep 15 20:42:16.973607 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.973420 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c6429a9c-1243-4d8d-a553-b971b86fe035" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:42:16.973677 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.973643 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-certificates\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:42:16.975640 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.975605 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c6429a9c-1243-4d8d-a553-b971b86fe035" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:42:16.975875 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.975832 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c6429a9c-1243-4d8d-a553-b971b86fe035" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:42:16.975994 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.975943 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c6429a9c-1243-4d8d-a553-b971b86fe035" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:42:16.976247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.976219 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c6429a9c-1243-4d8d-a553-b971b86fe035" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:42:16.976324 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.976293 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-kube-api-access-c2gbc" (OuterVolumeSpecName: "kube-api-access-c2gbc") pod "c6429a9c-1243-4d8d-a553-b971b86fe035" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035"). InnerVolumeSpecName "kube-api-access-c2gbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:42:16.976380 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.976325 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c6429a9c-1243-4d8d-a553-b971b86fe035" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:42:16.983138 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:16.983114 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6429a9c-1243-4d8d-a553-b971b86fe035-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c6429a9c-1243-4d8d-a553-b971b86fe035" (UID: "c6429a9c-1243-4d8d-a553-b971b86fe035"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:42:17.074704 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.074672 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-installation-pull-secrets\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:42:17.074704 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.074701 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6429a9c-1243-4d8d-a553-b971b86fe035-trusted-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:42:17.074704 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.074711 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6429a9c-1243-4d8d-a553-b971b86fe035-ca-trust-extracted\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:42:17.074932 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.074720 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6429a9c-1243-4d8d-a553-b971b86fe035-image-registry-private-configuration\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:42:17.074932 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.074729 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-registry-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:42:17.074932 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.074738 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2gbc\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-kube-api-access-c2gbc\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:42:17.074932 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.074746 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6429a9c-1243-4d8d-a553-b971b86fe035-bound-sa-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:42:17.382002 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.381965 2579 generic.go:356] "Generic (PLEG): container finished" podID="c6429a9c-1243-4d8d-a553-b971b86fe035" containerID="592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2" exitCode=0 Sep 15 20:42:17.382164 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.382019 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" Sep 15 20:42:17.382164 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.382046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" event={"ID":"c6429a9c-1243-4d8d-a553-b971b86fe035","Type":"ContainerDied","Data":"592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2"} Sep 15 20:42:17.382164 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.382099 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646ff6c755-6br8d" event={"ID":"c6429a9c-1243-4d8d-a553-b971b86fe035","Type":"ContainerDied","Data":"17e22081a9825b7f5482b322027ef681b6763a8615aedc3aa0d6a3666c43c1d7"} Sep 15 20:42:17.382164 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.382127 2579 scope.go:117] "RemoveContainer" containerID="592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2" Sep 15 20:42:17.391376 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.391358 2579 scope.go:117] "RemoveContainer" containerID="592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2" Sep 15 20:42:17.391704 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:42:17.391680 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2\": container with ID starting with 592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2 not found: ID does not exist" containerID="592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2" Sep 15 20:42:17.391779 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.391717 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2"} err="failed to get container status \"592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2\": rpc error: code = NotFound desc = could not find container \"592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2\": container with ID starting with 592d864e3a7d8350a59a9c0ce4c058896bcd823150bfa8c5e4c4009b270271a2 not found: ID does not exist" Sep 15 20:42:17.398236 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.398211 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-646ff6c755-6br8d"] Sep 15 20:42:17.401943 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:17.401924 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-646ff6c755-6br8d"] Sep 15 20:42:19.310229 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:19.310194 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6429a9c-1243-4d8d-a553-b971b86fe035" path="/var/lib/kubelet/pods/c6429a9c-1243-4d8d-a553-b971b86fe035/volumes" Sep 15 20:42:29.090159 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:29.090129 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:42:29.093918 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:29.093896 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d9d59fdf9-4kj7g" Sep 15 20:42:35.449454 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:35.449418 2579 generic.go:356] "Generic (PLEG): container finished" podID="167ff948-f63b-42d8-b67e-882ddafb2a0d" containerID="482f0a8d005ddf96bfb86463b60040a726f06285a56583eff92d224e6a98868d" exitCode=0 Sep 15 20:42:35.449915 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:35.449511 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" event={"ID":"167ff948-f63b-42d8-b67e-882ddafb2a0d","Type":"ContainerDied","Data":"482f0a8d005ddf96bfb86463b60040a726f06285a56583eff92d224e6a98868d"} Sep 15 20:42:35.449915 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:35.449834 2579 scope.go:117] "RemoveContainer" containerID="482f0a8d005ddf96bfb86463b60040a726f06285a56583eff92d224e6a98868d" Sep 15 20:42:35.451230 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:35.451209 2579 generic.go:356] "Generic (PLEG): container finished" podID="64f3e5db-c556-4406-827d-b1401e9ce442" containerID="aa2352d0407e11329e1c135668c8a5127b4ad0bec20027804bc5f6eec9cf534a" exitCode=0 Sep 15 20:42:35.451305 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:35.451257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" event={"ID":"64f3e5db-c556-4406-827d-b1401e9ce442","Type":"ContainerDied","Data":"aa2352d0407e11329e1c135668c8a5127b4ad0bec20027804bc5f6eec9cf534a"} Sep 15 20:42:35.451575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:35.451560 2579 scope.go:117] "RemoveContainer" containerID="aa2352d0407e11329e1c135668c8a5127b4ad0bec20027804bc5f6eec9cf534a" Sep 15 20:42:36.462564 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:36.462526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m8jgr" event={"ID":"64f3e5db-c556-4406-827d-b1401e9ce442","Type":"ContainerStarted","Data":"a8b9d7e554e55f12ce8329fe43b50c550883d86168ce0eaf0381f002c222b947"} Sep 15 20:42:36.464830 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:36.464806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wfzch" event={"ID":"167ff948-f63b-42d8-b67e-882ddafb2a0d","Type":"ContainerStarted","Data":"f4f32b77b8f885ef7de63437589943f7ab0f74bebcb6eb3b1c7cc84b2e55b9a0"} Sep 15 20:42:50.796720 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:50.796685 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:42:50.816089 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:50.816059 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:42:51.542382 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:51.542353 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:42:57.306717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:57.306678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:42:57.312107 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:57.309920 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11e01e0e-b7c2-41fb-8152-77fdd5533940-metrics-certs\") pod \"network-metrics-daemon-d628s\" (UID: \"11e01e0e-b7c2-41fb-8152-77fdd5533940\") " pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:42:57.609481 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:57.609391 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nxplx\"" Sep 15 20:42:57.617586 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:57.617559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d628s" Sep 15 20:42:57.887812 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:57.887740 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d628s"] Sep 15 20:42:57.891936 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:42:57.891908 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e01e0e_b7c2_41fb_8152_77fdd5533940.slice/crio-3ab1dc84aa5830f2d1924fbd4e52d9d264a2c6a67ebc9044e61dfcaa98e8e790 WatchSource:0}: Error finding container 3ab1dc84aa5830f2d1924fbd4e52d9d264a2c6a67ebc9044e61dfcaa98e8e790: Status 404 returned error can't find the container with id 3ab1dc84aa5830f2d1924fbd4e52d9d264a2c6a67ebc9044e61dfcaa98e8e790 Sep 15 20:42:58.553018 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:58.552985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d628s" event={"ID":"11e01e0e-b7c2-41fb-8152-77fdd5533940","Type":"ContainerStarted","Data":"3ab1dc84aa5830f2d1924fbd4e52d9d264a2c6a67ebc9044e61dfcaa98e8e790"} Sep 15 20:42:59.558454 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:59.558419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d628s" event={"ID":"11e01e0e-b7c2-41fb-8152-77fdd5533940","Type":"ContainerStarted","Data":"4bf1107003e88cfc393e80df092dd38eb02d4d59a5932c2f9efabbf7f2d12331"} Sep 15 20:42:59.558860 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:59.558480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d628s" event={"ID":"11e01e0e-b7c2-41fb-8152-77fdd5533940","Type":"ContainerStarted","Data":"53dd5f5ae6bea0a642e0b0e8516ad3a6d212c321d607ef67340e3a8484e49835"} Sep 15 20:42:59.574285 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:42:59.574238 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d628s" podStartSLOduration=374.249391815 podStartE2EDuration="6m15.574222557s" totalImagesPullingTime="1.324830742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:42:59.572765603 +0000 UTC m=+376.766719058" watchObservedRunningTime="2026-09-15 20:42:59.574222557 +0000 UTC m=+376.768176048" Sep 15 20:43:08.882842 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:08.882810 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 15 20:43:08.883247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:08.883079 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="prometheus" containerID="cri-o://696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08" gracePeriod=600 Sep 15 20:43:08.883247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:08.883108 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy" containerID="cri-o://57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70" gracePeriod=600 Sep 15 20:43:08.883247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:08.883137 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="thanos-sidecar" containerID="cri-o://0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2" gracePeriod=600 Sep 15 20:43:08.883247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:08.883202 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e" gracePeriod=600 Sep 15 20:43:08.883247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:08.883205 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy-web" containerID="cri-o://3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979" gracePeriod=600 Sep 15 20:43:08.883247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:08.883226 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="config-reloader" containerID="cri-o://f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9" gracePeriod=600 Sep 15 20:43:09.115910 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.115887 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.202456 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202368 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-serving-certs-ca-bundle\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202456 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202404 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v8qv\" (UniqueName: \"kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-kube-api-access-7v8qv\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202456 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202428 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-tls\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202456 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202445 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-metrics-client-certs\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202485 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-thanos-prometheus-http-client-file\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202524 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-2\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202553 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202596 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-config\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202623 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-kube-rbac-proxy\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202648 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-db\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202672 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202697 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-metrics-client-ca\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202726 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-grpc-tls\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202755 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-config-out\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.202804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202793 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-kubelet-serving-ca-bundle\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.203288 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202816 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-web-config\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.203288 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:09.203288 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202848 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-tls-assets\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.203288 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202877 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-0\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.203288 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202916 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-trusted-ca-bundle\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.203288 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.202974 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-1\") pod \"4e2539cf-5689-42bb-996a-189f35a62da8\" (UID: \"4e2539cf-5689-42bb-996a-189f35a62da8\") " Sep 15 20:43:09.203601 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.203342 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:09.204272 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.203873 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:43:09.204764 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.204738 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.204886 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.204772 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.204886 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.204788 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-db\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.205719 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.205411 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:09.206338 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.206160 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:09.207121 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.206785 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:09.207121 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.206896 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:09.207121 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.206925 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-config-out" (OuterVolumeSpecName: "config-out") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:43:09.207121 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.206956 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:09.207121 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.207117 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-kube-api-access-7v8qv" (OuterVolumeSpecName: "kube-api-access-7v8qv") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "kube-api-access-7v8qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:43:09.207451 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.207118 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:09.207451 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.207180 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:09.207451 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.207322 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-config" (OuterVolumeSpecName: "config") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:09.207675 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.207646 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:09.208473 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.208441 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:09.208751 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.208698 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:09.209013 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.208812 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:43:09.209013 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.208894 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:09.209013 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.209009 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:09.220351 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.220326 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-web-config" (OuterVolumeSpecName: "web-config") pod "4e2539cf-5689-42bb-996a-189f35a62da8" (UID: "4e2539cf-5689-42bb-996a-189f35a62da8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:09.306051 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306021 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8qv\" (UniqueName: \"kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-kube-api-access-7v8qv\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306051 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306047 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306051 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306056 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-metrics-client-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306066 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306075 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306084 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306093 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306102 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-kube-rbac-proxy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306111 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306119 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-metrics-client-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306127 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-secret-grpc-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306135 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2539cf-5689-42bb-996a-189f35a62da8-config-out\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306142 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306150 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2539cf-5689-42bb-996a-189f35a62da8-web-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306158 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2539cf-5689-42bb-996a-189f35a62da8-tls-assets\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306166 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.306247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.306176 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2539cf-5689-42bb-996a-189f35a62da8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:43:09.605696 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605656 2579 generic.go:356] "Generic (PLEG): container finished" podID="4e2539cf-5689-42bb-996a-189f35a62da8" containerID="a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e" exitCode=0 Sep 15 20:43:09.605696 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605684 2579 generic.go:356] "Generic (PLEG): container finished" podID="4e2539cf-5689-42bb-996a-189f35a62da8" containerID="57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70" exitCode=0 Sep 15 20:43:09.605696 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605696 2579 generic.go:356] "Generic (PLEG): container finished" podID="4e2539cf-5689-42bb-996a-189f35a62da8" containerID="3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979" exitCode=0 Sep 15 20:43:09.605920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605705 2579 generic.go:356] "Generic (PLEG): container finished" podID="4e2539cf-5689-42bb-996a-189f35a62da8" containerID="0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2" exitCode=0 Sep 15 20:43:09.605920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605712 2579 generic.go:356] "Generic (PLEG): container finished" podID="4e2539cf-5689-42bb-996a-189f35a62da8" containerID="f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9" exitCode=0 Sep 15 20:43:09.605920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605720 2579 generic.go:356] "Generic (PLEG): container finished" podID="4e2539cf-5689-42bb-996a-189f35a62da8" containerID="696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08" exitCode=0 Sep 15 20:43:09.605920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605790 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerDied","Data":"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e"} Sep 15 20:43:09.605920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605842 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerDied","Data":"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70"} Sep 15 20:43:09.605920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605793 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.605920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerDied","Data":"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979"} Sep 15 20:43:09.605920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605905 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerDied","Data":"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2"} Sep 15 20:43:09.606260 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605930 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerDied","Data":"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9"} Sep 15 20:43:09.606260 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605949 2579 scope.go:117] "RemoveContainer" containerID="a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e" Sep 15 20:43:09.606260 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605963 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerDied","Data":"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08"} Sep 15 20:43:09.606260 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.605988 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e2539cf-5689-42bb-996a-189f35a62da8","Type":"ContainerDied","Data":"c617a4839cfdc8eca0938c4ae1157d826e24da75700c1997fe961ff3b6c895e6"} Sep 15 20:43:09.616020 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.615989 2579 scope.go:117] "RemoveContainer" containerID="57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70" Sep 15 20:43:09.623542 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.623524 2579 scope.go:117] "RemoveContainer" containerID="3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979" Sep 15 20:43:09.630094 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.630059 2579 scope.go:117] "RemoveContainer" containerID="0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2" Sep 15 20:43:09.632480 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.632461 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 15 20:43:09.637120 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.637089 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 15 20:43:09.638037 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.638022 2579 scope.go:117] "RemoveContainer" containerID="f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9" Sep 15 20:43:09.644897 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.644878 2579 scope.go:117] "RemoveContainer" containerID="696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08" Sep 15 20:43:09.652241 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.652221 2579 scope.go:117] "RemoveContainer" containerID="a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590" Sep 15 20:43:09.659232 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.659217 2579 scope.go:117] "RemoveContainer" containerID="a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e" Sep 15 20:43:09.659535 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:43:09.659480 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": container with ID starting with a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e not found: ID does not exist" containerID="a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e" Sep 15 20:43:09.659632 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.659528 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e"} err="failed to get container status \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": rpc error: code = NotFound desc = could not find container \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": container with ID starting with a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e not found: ID does not exist" Sep 15 20:43:09.659632 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.659549 2579 scope.go:117] "RemoveContainer" containerID="57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70" Sep 15 20:43:09.659807 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:43:09.659792 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": container with ID starting with 57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70 not found: ID does not exist" containerID="57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70" Sep 15 20:43:09.659848 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.659809 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70"} err="failed to get container status \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": rpc error: code = NotFound desc = could not find container \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": container with ID starting with 57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70 not found: ID does not exist" Sep 15 20:43:09.659848 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.659821 2579 scope.go:117] "RemoveContainer" containerID="3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979" Sep 15 20:43:09.660079 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:43:09.660055 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": container with ID starting with 3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979 not found: ID does not exist" containerID="3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979" Sep 15 20:43:09.660207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.660082 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979"} err="failed to get container status \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": rpc error: code = NotFound desc = could not find container \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": container with ID starting with 3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979 not found: ID does not exist" Sep 15 20:43:09.660207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.660096 2579 scope.go:117] "RemoveContainer" containerID="0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2" Sep 15 20:43:09.660598 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:43:09.660432 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": container with ID starting with 0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2 not found: ID does not exist" containerID="0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2" Sep 15 20:43:09.660598 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.660463 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2"} err="failed to get container status \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": rpc error: code = NotFound desc = could not find container \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": container with ID starting with 0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2 not found: ID does not exist" Sep 15 20:43:09.660598 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.660546 2579 scope.go:117] "RemoveContainer" containerID="f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9" Sep 15 20:43:09.661006 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:43:09.660975 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": container with ID starting with f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9 not found: ID does not exist" containerID="f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9" Sep 15 20:43:09.661093 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.661013 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9"} err="failed to get container status \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": rpc error: code = NotFound desc = could not find container \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": container with ID starting with f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9 not found: ID does not exist" Sep 15 20:43:09.661093 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.661032 2579 scope.go:117] "RemoveContainer" containerID="696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08" Sep 15 20:43:09.661756 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:43:09.661731 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": container with ID starting with 696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08 not found: ID does not exist" containerID="696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08" Sep 15 20:43:09.661826 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.661761 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08"} err="failed to get container status \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": rpc error: code = NotFound desc = could not find container \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": container with ID starting with 696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08 not found: ID does not exist" Sep 15 20:43:09.661826 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.661780 2579 scope.go:117] "RemoveContainer" containerID="a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590" Sep 15 20:43:09.662075 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:43:09.662059 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": container with ID starting with a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590 not found: ID does not exist" containerID="a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590" Sep 15 20:43:09.662130 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662078 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590"} err="failed to get container status \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": rpc error: code = NotFound desc = could not find container \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": container with ID starting with a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590 not found: ID does not exist" Sep 15 20:43:09.662130 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662096 2579 scope.go:117] "RemoveContainer" containerID="a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e" Sep 15 20:43:09.662350 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662332 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 15 20:43:09.662394 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662353 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e"} err="failed to get container status \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": rpc error: code = NotFound desc = could not find container \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": container with ID starting with a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e not found: ID does not exist" Sep 15 20:43:09.662394 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662370 2579 scope.go:117] "RemoveContainer" containerID="57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70" Sep 15 20:43:09.662680 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662648 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70"} err="failed to get container status \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": rpc error: code = NotFound desc = could not find container \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": container with ID starting with 57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70 not found: ID does not exist" Sep 15 20:43:09.662752 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662683 2579 scope.go:117] "RemoveContainer" containerID="3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979" Sep 15 20:43:09.662752 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662684 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="init-config-reloader" Sep 15 20:43:09.662752 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662699 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="init-config-reloader" Sep 15 20:43:09.662752 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662719 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="thanos-sidecar" Sep 15 20:43:09.662752 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662727 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="thanos-sidecar" Sep 15 20:43:09.662752 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662745 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="config-reloader" Sep 15 20:43:09.662752 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662753 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="config-reloader" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662761 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy-web" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662766 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy-web" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662772 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6429a9c-1243-4d8d-a553-b971b86fe035" containerName="registry" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662777 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6429a9c-1243-4d8d-a553-b971b86fe035" containerName="registry" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662785 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy-thanos" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662790 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy-thanos" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662798 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="prometheus" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662803 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="prometheus" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662810 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662818 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662873 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662882 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6429a9c-1243-4d8d-a553-b971b86fe035" containerName="registry" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662887 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy-thanos" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662896 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="config-reloader" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662907 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="prometheus" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662914 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="thanos-sidecar" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662919 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" containerName="kube-rbac-proxy-web" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662914 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979"} err="failed to get container status \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": rpc error: code = NotFound desc = could not find container \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": container with ID starting with 3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979 not found: ID does not exist" Sep 15 20:43:09.662992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.662931 2579 scope.go:117] "RemoveContainer" containerID="0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2" Sep 15 20:43:09.663672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.663129 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2"} err="failed to get container status \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": rpc error: code = NotFound desc = could not find container \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": container with ID starting with 0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2 not found: ID does not exist" Sep 15 20:43:09.663672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.663143 2579 scope.go:117] "RemoveContainer" containerID="f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9" Sep 15 20:43:09.663672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.663326 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9"} err="failed to get container status \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": rpc error: code = NotFound desc = could not find container \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": container with ID starting with f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9 not found: ID does not exist" Sep 15 20:43:09.663672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.663338 2579 scope.go:117] "RemoveContainer" containerID="696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08" Sep 15 20:43:09.663672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.663553 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08"} err="failed to get container status \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": rpc error: code = NotFound desc = could not find container \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": container with ID starting with 696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08 not found: ID does not exist" Sep 15 20:43:09.663672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.663566 2579 scope.go:117] "RemoveContainer" containerID="a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590" Sep 15 20:43:09.663898 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.663808 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590"} err="failed to get container status \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": rpc error: code = NotFound desc = could not find container \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": container with ID starting with a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590 not found: ID does not exist" Sep 15 20:43:09.663898 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.663834 2579 scope.go:117] "RemoveContainer" containerID="a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e" Sep 15 20:43:09.664046 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.664029 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e"} err="failed to get container status \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": rpc error: code = NotFound desc = could not find container \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": container with ID starting with a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e not found: ID does not exist" Sep 15 20:43:09.664090 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.664046 2579 scope.go:117] "RemoveContainer" containerID="57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70" Sep 15 20:43:09.664277 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.664250 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70"} err="failed to get container status \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": rpc error: code = NotFound desc = could not find container \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": container with ID starting with 57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70 not found: ID does not exist" Sep 15 20:43:09.664329 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.664278 2579 scope.go:117] "RemoveContainer" containerID="3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979" Sep 15 20:43:09.664549 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.664530 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979"} err="failed to get container status \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": rpc error: code = NotFound desc = could not find container \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": container with ID starting with 3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979 not found: ID does not exist" Sep 15 20:43:09.664628 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.664551 2579 scope.go:117] "RemoveContainer" containerID="0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2" Sep 15 20:43:09.664775 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.664760 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2"} err="failed to get container status \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": rpc error: code = NotFound desc = could not find container \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": container with ID starting with 0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2 not found: ID does not exist" Sep 15 20:43:09.664823 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.664783 2579 scope.go:117] "RemoveContainer" containerID="f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9" Sep 15 20:43:09.664999 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.664963 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9"} err="failed to get container status \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": rpc error: code = NotFound desc = could not find container \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": container with ID starting with f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9 not found: ID does not exist" Sep 15 20:43:09.664999 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.664977 2579 scope.go:117] "RemoveContainer" containerID="696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08" Sep 15 20:43:09.665185 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.665170 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08"} err="failed to get container status \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": rpc error: code = NotFound desc = could not find container \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": container with ID starting with 696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08 not found: ID does not exist" Sep 15 20:43:09.665233 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.665187 2579 scope.go:117] "RemoveContainer" containerID="a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590" Sep 15 20:43:09.665432 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.665412 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590"} err="failed to get container status \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": rpc error: code = NotFound desc = could not find container \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": container with ID starting with a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590 not found: ID does not exist" Sep 15 20:43:09.665479 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.665433 2579 scope.go:117] "RemoveContainer" containerID="a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e" Sep 15 20:43:09.665685 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.665669 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e"} err="failed to get container status \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": rpc error: code = NotFound desc = could not find container \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": container with ID starting with a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e not found: ID does not exist" Sep 15 20:43:09.665742 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.665685 2579 scope.go:117] "RemoveContainer" containerID="57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70" Sep 15 20:43:09.665877 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.665862 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70"} err="failed to get container status \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": rpc error: code = NotFound desc = could not find container \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": container with ID starting with 57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70 not found: ID does not exist" Sep 15 20:43:09.665910 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.665877 2579 scope.go:117] "RemoveContainer" containerID="3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979" Sep 15 20:43:09.666078 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.666065 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979"} err="failed to get container status \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": rpc error: code = NotFound desc = could not find container \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": container with ID starting with 3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979 not found: ID does not exist" Sep 15 20:43:09.666115 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.666079 2579 scope.go:117] "RemoveContainer" containerID="0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2" Sep 15 20:43:09.666281 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.666266 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2"} err="failed to get container status \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": rpc error: code = NotFound desc = could not find container \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": container with ID starting with 0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2 not found: ID does not exist" Sep 15 20:43:09.666348 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.666280 2579 scope.go:117] "RemoveContainer" containerID="f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9" Sep 15 20:43:09.666544 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.666524 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9"} err="failed to get container status \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": rpc error: code = NotFound desc = could not find container \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": container with ID starting with f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9 not found: ID does not exist" Sep 15 20:43:09.666591 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.666545 2579 scope.go:117] "RemoveContainer" containerID="696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08" Sep 15 20:43:09.666754 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.666739 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08"} err="failed to get container status \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": rpc error: code = NotFound desc = could not find container \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": container with ID starting with 696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08 not found: ID does not exist" Sep 15 20:43:09.666795 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.666754 2579 scope.go:117] "RemoveContainer" containerID="a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590" Sep 15 20:43:09.666964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.666947 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590"} err="failed to get container status \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": rpc error: code = NotFound desc = could not find container \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": container with ID starting with a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590 not found: ID does not exist" Sep 15 20:43:09.666964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.666963 2579 scope.go:117] "RemoveContainer" containerID="a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e" Sep 15 20:43:09.667147 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.667132 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e"} err="failed to get container status \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": rpc error: code = NotFound desc = could not find container \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": container with ID starting with a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e not found: ID does not exist" Sep 15 20:43:09.667187 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.667146 2579 scope.go:117] "RemoveContainer" containerID="57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70" Sep 15 20:43:09.667336 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.667321 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70"} err="failed to get container status \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": rpc error: code = NotFound desc = could not find container \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": container with ID starting with 57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70 not found: ID does not exist" Sep 15 20:43:09.667390 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.667337 2579 scope.go:117] "RemoveContainer" containerID="3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979" Sep 15 20:43:09.667578 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.667562 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979"} err="failed to get container status \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": rpc error: code = NotFound desc = could not find container \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": container with ID starting with 3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979 not found: ID does not exist" Sep 15 20:43:09.667638 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.667579 2579 scope.go:117] "RemoveContainer" containerID="0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2" Sep 15 20:43:09.667794 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.667774 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2"} err="failed to get container status \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": rpc error: code = NotFound desc = could not find container \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": container with ID starting with 0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2 not found: ID does not exist" Sep 15 20:43:09.667794 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.667791 2579 scope.go:117] "RemoveContainer" containerID="f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9" Sep 15 20:43:09.667931 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.667915 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.668000 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.667979 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9"} err="failed to get container status \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": rpc error: code = NotFound desc = could not find container \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": container with ID starting with f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9 not found: ID does not exist" Sep 15 20:43:09.668061 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.668000 2579 scope.go:117] "RemoveContainer" containerID="696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08" Sep 15 20:43:09.668303 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.668277 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08"} err="failed to get container status \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": rpc error: code = NotFound desc = could not find container \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": container with ID starting with 696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08 not found: ID does not exist" Sep 15 20:43:09.668382 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.668304 2579 scope.go:117] "RemoveContainer" containerID="a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590" Sep 15 20:43:09.668675 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.668654 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590"} err="failed to get container status \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": rpc error: code = NotFound desc = could not find container \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": container with ID starting with a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590 not found: ID does not exist" Sep 15 20:43:09.668675 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.668674 2579 scope.go:117] "RemoveContainer" containerID="a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e" Sep 15 20:43:09.669000 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.668981 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e"} err="failed to get container status \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": rpc error: code = NotFound desc = could not find container \"a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e\": container with ID starting with a1fd7060ee0dee52b25bc8e474254fd1fef15b8245c28f3165580ac31bd4e55e not found: ID does not exist" Sep 15 20:43:09.669051 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.669001 2579 scope.go:117] "RemoveContainer" containerID="57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70" Sep 15 20:43:09.669271 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.669249 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70"} err="failed to get container status \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": rpc error: code = NotFound desc = could not find container \"57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70\": container with ID starting with 57b91622d3eb98cba080a8d5ce6ec122a0d49150a1f20e108f193d6e49aeed70 not found: ID does not exist" Sep 15 20:43:09.669342 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.669272 2579 scope.go:117] "RemoveContainer" containerID="3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979" Sep 15 20:43:09.669579 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.669561 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979"} err="failed to get container status \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": rpc error: code = NotFound desc = could not find container \"3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979\": container with ID starting with 3f7558e3bf0a01997f813262deb8b2ba025abfa7f07afee7d263133d021e5979 not found: ID does not exist" Sep 15 20:43:09.669632 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.669580 2579 scope.go:117] "RemoveContainer" containerID="0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2" Sep 15 20:43:09.669846 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.669819 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2"} err="failed to get container status \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": rpc error: code = NotFound desc = could not find container \"0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2\": container with ID starting with 0b18d8a40d193889e4d2154289e0fe000e081690c5ee099991965938fe12cfc2 not found: ID does not exist" Sep 15 20:43:09.669846 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.669845 2579 scope.go:117] "RemoveContainer" containerID="f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9" Sep 15 20:43:09.670074 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670052 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9"} err="failed to get container status \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": rpc error: code = NotFound desc = could not find container \"f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9\": container with ID starting with f4da480056b92d1f972f9db17ac225df40f4cdff4075a30e63011964f47902b9 not found: ID does not exist" Sep 15 20:43:09.670121 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670077 2579 scope.go:117] "RemoveContainer" containerID="696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08" Sep 15 20:43:09.670344 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670311 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08"} err="failed to get container status \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": rpc error: code = NotFound desc = could not find container \"696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08\": container with ID starting with 696442281bc0ad0fed0cc4fbda464584640e2e35bf3ddfe08dfbb27c5b5aee08 not found: ID does not exist" Sep 15 20:43:09.670344 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670333 2579 scope.go:117] "RemoveContainer" containerID="a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590" Sep 15 20:43:09.670480 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670451 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 15 20:43:09.670480 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670450 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7hbfm\"" Sep 15 20:43:09.670631 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670523 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 15 20:43:09.670739 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670716 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590"} err="failed to get container status \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": rpc error: code = NotFound desc = could not find container \"a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590\": container with ID starting with a4e2ec90b65f85224d9c5d775ecf153ea0579dd65162a05ad735a8c139b53590 not found: ID does not exist" Sep 15 20:43:09.670853 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670837 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 15 20:43:09.670942 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670909 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 15 20:43:09.670942 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670918 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 15 20:43:09.671072 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670973 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-epqppoagjaouo\"" Sep 15 20:43:09.671072 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670918 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 15 20:43:09.671072 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670946 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 15 20:43:09.671072 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.670970 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 15 20:43:09.671288 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.671270 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 15 20:43:09.671469 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.671453 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 15 20:43:09.671538 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.671490 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 15 20:43:09.673159 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.673142 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 15 20:43:09.674596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.674570 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 15 20:43:09.677420 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.677404 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 15 20:43:09.685484 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.685462 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 15 20:43:09.809770 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.809722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlkx4\" (UniqueName: \"kubernetes.io/projected/6470ac0a-6161-4495-bb61-5f67f4e2de14-kube-api-access-xlkx4\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.809770 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.809769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.809789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.809864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-config\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.809893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.809910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.809932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.809987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6470ac0a-6161-4495-bb61-5f67f4e2de14-config-out\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-web-config\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810270 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810643 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.810643 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.810296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6470ac0a-6161-4495-bb61-5f67f4e2de14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.910965 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.910940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-config\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.910968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.910983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6470ac0a-6161-4495-bb61-5f67f4e2de14-config-out\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-web-config\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6470ac0a-6161-4495-bb61-5f67f4e2de14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.911332 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkx4\" (UniqueName: \"kubernetes.io/projected/6470ac0a-6161-4495-bb61-5f67f4e2de14-kube-api-access-xlkx4\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.912170 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.912170 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.912170 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911735 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.912170 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.911771 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.912170 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.912108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.914370 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.914253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.914745 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.914399 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6470ac0a-6161-4495-bb61-5f67f4e2de14-config-out\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.914745 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.914407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.914745 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.914656 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-web-config\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.914745 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.914674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.914745 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.914681 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.915032 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.915013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-config\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.915687 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.915083 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.915687 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.915089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.915687 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.915384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.915687 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.915580 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.915973 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.915812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.917153 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.917131 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6470ac0a-6161-4495-bb61-5f67f4e2de14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.917548 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.917528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.917595 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.917537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6470ac0a-6161-4495-bb61-5f67f4e2de14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.917779 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.917760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6470ac0a-6161-4495-bb61-5f67f4e2de14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.923738 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.923716 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkx4\" (UniqueName: \"kubernetes.io/projected/6470ac0a-6161-4495-bb61-5f67f4e2de14-kube-api-access-xlkx4\") pod \"prometheus-k8s-0\" (UID: \"6470ac0a-6161-4495-bb61-5f67f4e2de14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:09.977818 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:09.977779 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:43:10.259693 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:10.259666 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 15 20:43:10.260580 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:43:10.260554 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6470ac0a_6161_4495_bb61_5f67f4e2de14.slice/crio-f4dfa730716e83fd5f15c08be5103d6f66d6d754a19d2d1cbaac0206659d5466 WatchSource:0}: Error finding container f4dfa730716e83fd5f15c08be5103d6f66d6d754a19d2d1cbaac0206659d5466: Status 404 returned error can't find the container with id f4dfa730716e83fd5f15c08be5103d6f66d6d754a19d2d1cbaac0206659d5466 Sep 15 20:43:10.611071 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:10.611037 2579 generic.go:356] "Generic (PLEG): container finished" podID="6470ac0a-6161-4495-bb61-5f67f4e2de14" containerID="0dbcd856b714b626adcb6ea08331c5e8ddd2627212634ee98c4b86ae549d9d1a" exitCode=0 Sep 15 20:43:10.611261 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:10.611127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6470ac0a-6161-4495-bb61-5f67f4e2de14","Type":"ContainerDied","Data":"0dbcd856b714b626adcb6ea08331c5e8ddd2627212634ee98c4b86ae549d9d1a"} Sep 15 20:43:10.611261 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:10.611210 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6470ac0a-6161-4495-bb61-5f67f4e2de14","Type":"ContainerStarted","Data":"f4dfa730716e83fd5f15c08be5103d6f66d6d754a19d2d1cbaac0206659d5466"} Sep 15 20:43:11.311008 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:11.310971 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2539cf-5689-42bb-996a-189f35a62da8" path="/var/lib/kubelet/pods/4e2539cf-5689-42bb-996a-189f35a62da8/volumes" Sep 15 20:43:11.619868 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:11.619780 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6470ac0a-6161-4495-bb61-5f67f4e2de14","Type":"ContainerStarted","Data":"66237d72f5c0e3d8740b17cac5a95f44c454b2c1eb896ef9463ed18480d68ee4"} Sep 15 20:43:11.619868 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:11.619851 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6470ac0a-6161-4495-bb61-5f67f4e2de14","Type":"ContainerStarted","Data":"a6760079589f663f4b76d70a33709e94649f046caf3020c33c59627e5f49de43"} Sep 15 20:43:11.620081 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:11.619896 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6470ac0a-6161-4495-bb61-5f67f4e2de14","Type":"ContainerStarted","Data":"b8a80d18357400e1da1527f2ea00f080372a405426d29bc5f40fe0745fcceecd"} Sep 15 20:43:11.620081 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:11.619929 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6470ac0a-6161-4495-bb61-5f67f4e2de14","Type":"ContainerStarted","Data":"1865f65c30445fc5bb1c31940dd3924faa5376214ebeba3b9c9cc26610d01231"} Sep 15 20:43:11.620081 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:11.619966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6470ac0a-6161-4495-bb61-5f67f4e2de14","Type":"ContainerStarted","Data":"a798e81a493b1fb5004c97d186fe3d50dc8bbf97b3dbe27e6687b8c65ff002d6"} Sep 15 20:43:11.620081 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:11.620005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6470ac0a-6161-4495-bb61-5f67f4e2de14","Type":"ContainerStarted","Data":"2ab2a19e0461e8f41585b71f6e25966619712753eaeb7368f552ea51a1c3001c"} Sep 15 20:43:11.647598 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:11.647546 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.628407342 podStartE2EDuration="2.647529387s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.122045ms" isStatefulPod=false podCreationTimestamp="2026-09-15 20:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:43:11.645325342 +0000 UTC m=+388.839278800" watchObservedRunningTime="2026-09-15 20:43:11.647529387 +0000 UTC m=+388.841482842" Sep 15 20:43:14.978175 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:43:14.978134 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:44:09.978951 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:44:09.978865 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:44:09.994773 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:44:09.994747 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:44:10.859604 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:44:10.859573 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 15 20:45:59.041649 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.041613 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bppt9"] Sep 15 20:45:59.044964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.044946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" Sep 15 20:45:59.047310 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.047287 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 15 20:45:59.047451 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.047316 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-b7f4x\"" Sep 15 20:45:59.047451 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.047292 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 15 20:45:59.053438 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.053412 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bppt9"] Sep 15 20:45:59.130014 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.129981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgnj\" (UniqueName: \"kubernetes.io/projected/1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17-kube-api-access-8kgnj\") pod \"cert-manager-cainjector-559cdd8c87-bppt9\" (UID: \"1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" Sep 15 20:45:59.130014 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.130026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bppt9\" (UID: \"1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" Sep 15 20:45:59.230857 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.230819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgnj\" (UniqueName: \"kubernetes.io/projected/1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17-kube-api-access-8kgnj\") pod \"cert-manager-cainjector-559cdd8c87-bppt9\" (UID: \"1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" Sep 15 20:45:59.230857 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.230863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bppt9\" (UID: \"1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" Sep 15 20:45:59.238800 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.238766 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bppt9\" (UID: \"1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" Sep 15 20:45:59.239018 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.238999 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgnj\" (UniqueName: \"kubernetes.io/projected/1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17-kube-api-access-8kgnj\") pod \"cert-manager-cainjector-559cdd8c87-bppt9\" (UID: \"1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" Sep 15 20:45:59.365213 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.365126 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" Sep 15 20:45:59.636337 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:45:59.636269 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bppt9"] Sep 15 20:45:59.639220 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:45:59.639192 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ad6e0aa_87fc_4383_9fc7_9fb7e6d11a17.slice/crio-33239134fb2867d6bd1d539f0b6cbb13e158014801258e24866698e3b69f6a87 WatchSource:0}: Error finding container 33239134fb2867d6bd1d539f0b6cbb13e158014801258e24866698e3b69f6a87: Status 404 returned error can't find the container with id 33239134fb2867d6bd1d539f0b6cbb13e158014801258e24866698e3b69f6a87 Sep 15 20:46:00.254754 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:00.254711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" event={"ID":"1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17","Type":"ContainerStarted","Data":"33239134fb2867d6bd1d539f0b6cbb13e158014801258e24866698e3b69f6a87"} Sep 15 20:46:03.268570 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:03.268538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" event={"ID":"1ad6e0aa-87fc-4383-9fc7-9fb7e6d11a17","Type":"ContainerStarted","Data":"0944d64bb99e8011f1ac8a56ccd88f5a0f0250842fd120f8f3cfe2134b92c5bb"} Sep 15 20:46:03.283510 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:03.283455 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bppt9" podStartSLOduration=1.250319188 podStartE2EDuration="4.283439706s" totalImagesPullingTime="3.033120518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:03.282738231 +0000 UTC m=+560.476691686" watchObservedRunningTime="2026-09-15 20:46:03.283439706 +0000 UTC m=+560.477393161" Sep 15 20:46:30.765280 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.765244 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj"] Sep 15 20:46:30.768564 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.768543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:30.770910 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.770887 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 15 20:46:30.770910 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.770905 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:46:30.771061 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.770918 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 15 20:46:30.771784 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.771767 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 15 20:46:30.771861 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.771805 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 15 20:46:30.771927 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.771904 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-w6j86\"" Sep 15 20:46:30.778408 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.778386 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj"] Sep 15 20:46:30.901894 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.901862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/887dee56-230a-46cc-83c7-c943667b4799-manager-config\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:30.901894 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.901898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/887dee56-230a-46cc-83c7-c943667b4799-cert\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:30.902102 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.901970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/887dee56-230a-46cc-83c7-c943667b4799-metrics-cert\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:30.902102 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:30.902025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57q8\" (UniqueName: \"kubernetes.io/projected/887dee56-230a-46cc-83c7-c943667b4799-kube-api-access-d57q8\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:31.002791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.002755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/887dee56-230a-46cc-83c7-c943667b4799-cert\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:31.002980 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.002813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/887dee56-230a-46cc-83c7-c943667b4799-metrics-cert\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:31.002980 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.002845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57q8\" (UniqueName: \"kubernetes.io/projected/887dee56-230a-46cc-83c7-c943667b4799-kube-api-access-d57q8\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:31.002980 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.002869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/887dee56-230a-46cc-83c7-c943667b4799-manager-config\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:31.003644 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.003551 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/887dee56-230a-46cc-83c7-c943667b4799-manager-config\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:31.005485 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.005466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/887dee56-230a-46cc-83c7-c943667b4799-cert\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:31.005597 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.005517 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/887dee56-230a-46cc-83c7-c943667b4799-metrics-cert\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:31.010615 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.010594 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57q8\" (UniqueName: \"kubernetes.io/projected/887dee56-230a-46cc-83c7-c943667b4799-kube-api-access-d57q8\") pod \"lws-controller-manager-675cf9bd68-w7zjj\" (UID: \"887dee56-230a-46cc-83c7-c943667b4799\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:31.077707 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.077628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:31.349609 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.349583 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj"] Sep 15 20:46:31.352206 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:46:31.352173 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887dee56_230a_46cc_83c7_c943667b4799.slice/crio-eee53ecdbb305a8436abfe51f4b62a031ec67ddc605f227a4f306b61cd0760c0 WatchSource:0}: Error finding container eee53ecdbb305a8436abfe51f4b62a031ec67ddc605f227a4f306b61cd0760c0: Status 404 returned error can't find the container with id eee53ecdbb305a8436abfe51f4b62a031ec67ddc605f227a4f306b61cd0760c0 Sep 15 20:46:31.370418 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:31.370389 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" event={"ID":"887dee56-230a-46cc-83c7-c943667b4799","Type":"ContainerStarted","Data":"eee53ecdbb305a8436abfe51f4b62a031ec67ddc605f227a4f306b61cd0760c0"} Sep 15 20:46:32.020520 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.019000 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q"] Sep 15 20:46:32.024994 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.024971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.028113 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.028079 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 15 20:46:32.028113 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.028096 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 15 20:46:32.028337 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.028148 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 15 20:46:32.028383 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.028361 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fg66b\"" Sep 15 20:46:32.028726 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.028714 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 15 20:46:32.038483 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.038462 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q"] Sep 15 20:46:32.213555 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.213483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/42a75d67-bde1-4726-839b-cb9faa24cb0e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.213731 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.213646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ps88\" (UniqueName: \"kubernetes.io/projected/42a75d67-bde1-4726-839b-cb9faa24cb0e-kube-api-access-9ps88\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.213731 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.213701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/42a75d67-bde1-4726-839b-cb9faa24cb0e-manifests\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.213842 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.213731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42a75d67-bde1-4726-839b-cb9faa24cb0e-tmp\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.213842 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.213806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42a75d67-bde1-4726-839b-cb9faa24cb0e-webhook-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.315039 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.314518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/42a75d67-bde1-4726-839b-cb9faa24cb0e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.315039 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.314605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ps88\" (UniqueName: \"kubernetes.io/projected/42a75d67-bde1-4726-839b-cb9faa24cb0e-kube-api-access-9ps88\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.315039 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.314638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/42a75d67-bde1-4726-839b-cb9faa24cb0e-manifests\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.315039 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.314669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42a75d67-bde1-4726-839b-cb9faa24cb0e-tmp\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.315039 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.314732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42a75d67-bde1-4726-839b-cb9faa24cb0e-webhook-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.315421 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.315102 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/42a75d67-bde1-4726-839b-cb9faa24cb0e-manifests\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.318298 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.318250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/42a75d67-bde1-4726-839b-cb9faa24cb0e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.318485 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.318461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42a75d67-bde1-4726-839b-cb9faa24cb0e-webhook-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.318974 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.318944 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42a75d67-bde1-4726-839b-cb9faa24cb0e-tmp\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.324712 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.324688 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ps88\" (UniqueName: \"kubernetes.io/projected/42a75d67-bde1-4726-839b-cb9faa24cb0e-kube-api-access-9ps88\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-tm88q\" (UID: \"42a75d67-bde1-4726-839b-cb9faa24cb0e\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.335925 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.335704 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:32.630683 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:32.630643 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q"] Sep 15 20:46:32.633469 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:46:32.633435 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a75d67_bde1_4726_839b_cb9faa24cb0e.slice/crio-e0d8a60d20d16cfaa14f4061b8fbf37b1cd3f21aa3102af8df5def3234b67ab4 WatchSource:0}: Error finding container e0d8a60d20d16cfaa14f4061b8fbf37b1cd3f21aa3102af8df5def3234b67ab4: Status 404 returned error can't find the container with id e0d8a60d20d16cfaa14f4061b8fbf37b1cd3f21aa3102af8df5def3234b67ab4 Sep 15 20:46:33.382203 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:33.382161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" event={"ID":"42a75d67-bde1-4726-839b-cb9faa24cb0e","Type":"ContainerStarted","Data":"e0d8a60d20d16cfaa14f4061b8fbf37b1cd3f21aa3102af8df5def3234b67ab4"} Sep 15 20:46:34.389654 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:34.389618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" event={"ID":"887dee56-230a-46cc-83c7-c943667b4799","Type":"ContainerStarted","Data":"7db9c2bd4a3cec2ec3cc70265882d845aa3fbaecec565655a532c64c130e09cd"} Sep 15 20:46:34.390148 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:34.389794 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:34.408773 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:34.408706 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" podStartSLOduration=1.876678914 podStartE2EDuration="4.408678771s" totalImagesPullingTime="2.531999857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:34.407306487 +0000 UTC m=+591.601259942" watchObservedRunningTime="2026-09-15 20:46:34.408678771 +0000 UTC m=+591.602632226" Sep 15 20:46:37.404314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:37.404271 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" event={"ID":"42a75d67-bde1-4726-839b-cb9faa24cb0e","Type":"ContainerStarted","Data":"726c67bcc05ffa179ed3419c80a43e86f8fb092a7124ec07309c956224d7ae16"} Sep 15 20:46:38.409447 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:38.409413 2579 generic.go:356] "Generic (PLEG): container finished" podID="42a75d67-bde1-4726-839b-cb9faa24cb0e" containerID="726c67bcc05ffa179ed3419c80a43e86f8fb092a7124ec07309c956224d7ae16" exitCode=0 Sep 15 20:46:38.409852 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:38.409472 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" event={"ID":"42a75d67-bde1-4726-839b-cb9faa24cb0e","Type":"ContainerDied","Data":"726c67bcc05ffa179ed3419c80a43e86f8fb092a7124ec07309c956224d7ae16"} Sep 15 20:46:39.415366 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:39.415328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" event={"ID":"42a75d67-bde1-4726-839b-cb9faa24cb0e","Type":"ContainerStarted","Data":"6cee3e531d8eaf6cf37ba0411bd12090d7daea6001d3cd6127b49ed8dc3b5a53"} Sep 15 20:46:39.415772 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:39.415458 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:39.434647 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:39.434592 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" podStartSLOduration=3.108733849 podStartE2EDuration="8.43457155s" totalImagesPullingTime="4.251325132s" totalInitContainerRuntime="1.074512569s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:39.433907006 +0000 UTC m=+596.627860461" watchObservedRunningTime="2026-09-15 20:46:39.43457155 +0000 UTC m=+596.628525005" Sep 15 20:46:45.396423 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:45.396397 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-w7zjj" Sep 15 20:46:50.421201 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:50.421164 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-tm88q" Sep 15 20:46:52.464792 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.464758 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx"] Sep 15 20:46:52.468547 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.468523 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:52.471232 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.471186 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 15 20:46:52.471358 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.471311 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 15 20:46:52.472037 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.472018 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vgkd8\"" Sep 15 20:46:52.472037 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.472034 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 15 20:46:52.472229 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.472082 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 15 20:46:52.477832 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.477811 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx"] Sep 15 20:46:52.580670 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.580642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c512b4-9c74-4fec-8cac-427b40788b97-tls-certs\") pod \"kube-auth-proxy-65544b76bd-m4gxx\" (UID: \"08c512b4-9c74-4fec-8cac-427b40788b97\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:52.580831 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.580728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08c512b4-9c74-4fec-8cac-427b40788b97-tmp\") pod \"kube-auth-proxy-65544b76bd-m4gxx\" (UID: \"08c512b4-9c74-4fec-8cac-427b40788b97\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:52.580831 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.580776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf8qw\" (UniqueName: \"kubernetes.io/projected/08c512b4-9c74-4fec-8cac-427b40788b97-kube-api-access-rf8qw\") pod \"kube-auth-proxy-65544b76bd-m4gxx\" (UID: \"08c512b4-9c74-4fec-8cac-427b40788b97\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:52.681933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.681897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c512b4-9c74-4fec-8cac-427b40788b97-tls-certs\") pod \"kube-auth-proxy-65544b76bd-m4gxx\" (UID: \"08c512b4-9c74-4fec-8cac-427b40788b97\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:52.682108 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.681943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08c512b4-9c74-4fec-8cac-427b40788b97-tmp\") pod \"kube-auth-proxy-65544b76bd-m4gxx\" (UID: \"08c512b4-9c74-4fec-8cac-427b40788b97\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:52.682108 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.681963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf8qw\" (UniqueName: \"kubernetes.io/projected/08c512b4-9c74-4fec-8cac-427b40788b97-kube-api-access-rf8qw\") pod \"kube-auth-proxy-65544b76bd-m4gxx\" (UID: \"08c512b4-9c74-4fec-8cac-427b40788b97\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:52.684337 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.684308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08c512b4-9c74-4fec-8cac-427b40788b97-tmp\") pod \"kube-auth-proxy-65544b76bd-m4gxx\" (UID: \"08c512b4-9c74-4fec-8cac-427b40788b97\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:52.684445 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.684396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c512b4-9c74-4fec-8cac-427b40788b97-tls-certs\") pod \"kube-auth-proxy-65544b76bd-m4gxx\" (UID: \"08c512b4-9c74-4fec-8cac-427b40788b97\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:52.689438 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.689416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf8qw\" (UniqueName: \"kubernetes.io/projected/08c512b4-9c74-4fec-8cac-427b40788b97-kube-api-access-rf8qw\") pod \"kube-auth-proxy-65544b76bd-m4gxx\" (UID: \"08c512b4-9c74-4fec-8cac-427b40788b97\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:52.778464 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:52.778441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" Sep 15 20:46:53.052838 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:53.052760 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx"] Sep 15 20:46:53.055370 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:46:53.055339 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c512b4_9c74_4fec_8cac_427b40788b97.slice/crio-386c26a4ea4ee134b80928755c7b7d411e6a42b0b2947d4312df6c4ecd115a04 WatchSource:0}: Error finding container 386c26a4ea4ee134b80928755c7b7d411e6a42b0b2947d4312df6c4ecd115a04: Status 404 returned error can't find the container with id 386c26a4ea4ee134b80928755c7b7d411e6a42b0b2947d4312df6c4ecd115a04 Sep 15 20:46:53.057147 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:53.057127 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 20:46:53.476068 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:53.475985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" event={"ID":"08c512b4-9c74-4fec-8cac-427b40788b97","Type":"ContainerStarted","Data":"386c26a4ea4ee134b80928755c7b7d411e6a42b0b2947d4312df6c4ecd115a04"} Sep 15 20:46:56.492719 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:56.492681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" event={"ID":"08c512b4-9c74-4fec-8cac-427b40788b97","Type":"ContainerStarted","Data":"e6bc54620dfdaff4c5123a26116003cca25d387a3c91906014b060a37a120a9e"} Sep 15 20:46:56.508620 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:46:56.508408 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65544b76bd-m4gxx" podStartSLOduration=1.780570912 podStartE2EDuration="4.508388015s" totalImagesPullingTime="2.727817103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:56.507747436 +0000 UTC m=+613.701700890" watchObservedRunningTime="2026-09-15 20:46:56.508388015 +0000 UTC m=+613.702341469" Sep 15 20:48:22.998915 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:22.998880 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk"] Sep 15 20:48:23.002110 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.002093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:23.004160 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.004140 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-x6xg6\"" Sep 15 20:48:23.004271 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.004140 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 15 20:48:23.004959 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.004942 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 15 20:48:23.014680 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.014659 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk"] Sep 15 20:48:23.143583 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.143550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dv4f\" (UniqueName: \"kubernetes.io/projected/7abf7291-c178-4afe-95da-d3e132228be0-kube-api-access-9dv4f\") pod \"limitador-operator-controller-manager-74495c69f8-pwxbk\" (UID: \"7abf7291-c178-4afe-95da-d3e132228be0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:23.244983 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.244949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dv4f\" (UniqueName: \"kubernetes.io/projected/7abf7291-c178-4afe-95da-d3e132228be0-kube-api-access-9dv4f\") pod \"limitador-operator-controller-manager-74495c69f8-pwxbk\" (UID: \"7abf7291-c178-4afe-95da-d3e132228be0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:23.254852 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.254792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dv4f\" (UniqueName: \"kubernetes.io/projected/7abf7291-c178-4afe-95da-d3e132228be0-kube-api-access-9dv4f\") pod \"limitador-operator-controller-manager-74495c69f8-pwxbk\" (UID: \"7abf7291-c178-4afe-95da-d3e132228be0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:23.311822 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.311793 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:23.588167 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.588139 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk"] Sep 15 20:48:23.590411 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:48:23.590383 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abf7291_c178_4afe_95da_d3e132228be0.slice/crio-c1664d563791936b6d58c3c2b4410979ff8ab149a7af9f9713bff9522c343213 WatchSource:0}: Error finding container c1664d563791936b6d58c3c2b4410979ff8ab149a7af9f9713bff9522c343213: Status 404 returned error can't find the container with id c1664d563791936b6d58c3c2b4410979ff8ab149a7af9f9713bff9522c343213 Sep 15 20:48:23.852710 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:23.852617 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" event={"ID":"7abf7291-c178-4afe-95da-d3e132228be0","Type":"ContainerStarted","Data":"c1664d563791936b6d58c3c2b4410979ff8ab149a7af9f9713bff9522c343213"} Sep 15 20:48:26.869016 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:26.868982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" event={"ID":"7abf7291-c178-4afe-95da-d3e132228be0","Type":"ContainerStarted","Data":"3e2fe2734a3939470b72462e78a85d6917740fa04c9a92e6b3e48c1ff890b56c"} Sep 15 20:48:26.869410 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:26.869110 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:26.886074 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:26.886025 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" podStartSLOduration=2.488464487 podStartE2EDuration="4.886009833s" totalImagesPullingTime="2.397545346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:48:26.884461733 +0000 UTC m=+704.078415190" watchObservedRunningTime="2026-09-15 20:48:26.886009833 +0000 UTC m=+704.079963287" Sep 15 20:48:37.876410 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:37.876328 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:45.769120 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:45.769086 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45"] Sep 15 20:48:45.772342 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:45.772326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:45.774639 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:45.774622 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rp6ch\"" Sep 15 20:48:45.795854 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:45.795831 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45"] Sep 15 20:48:45.838377 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:45.838343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bb091c5-dc9e-4c16-a492-c94bce01faca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-5zv45\" (UID: \"6bb091c5-dc9e-4c16-a492-c94bce01faca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:45.838568 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:45.838394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7nrn\" (UniqueName: \"kubernetes.io/projected/6bb091c5-dc9e-4c16-a492-c94bce01faca-kube-api-access-k7nrn\") pod \"kuadrant-operator-controller-manager-5f685c496f-5zv45\" (UID: \"6bb091c5-dc9e-4c16-a492-c94bce01faca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:45.939445 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:45.939401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bb091c5-dc9e-4c16-a492-c94bce01faca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-5zv45\" (UID: \"6bb091c5-dc9e-4c16-a492-c94bce01faca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:45.939445 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:45.939450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7nrn\" (UniqueName: \"kubernetes.io/projected/6bb091c5-dc9e-4c16-a492-c94bce01faca-kube-api-access-k7nrn\") pod \"kuadrant-operator-controller-manager-5f685c496f-5zv45\" (UID: \"6bb091c5-dc9e-4c16-a492-c94bce01faca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:45.939810 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:45.939787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bb091c5-dc9e-4c16-a492-c94bce01faca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-5zv45\" (UID: \"6bb091c5-dc9e-4c16-a492-c94bce01faca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:45.948300 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:45.948273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7nrn\" (UniqueName: \"kubernetes.io/projected/6bb091c5-dc9e-4c16-a492-c94bce01faca-kube-api-access-k7nrn\") pod \"kuadrant-operator-controller-manager-5f685c496f-5zv45\" (UID: \"6bb091c5-dc9e-4c16-a492-c94bce01faca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:46.081615 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.081529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:46.354926 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.354902 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45"] Sep 15 20:48:46.356875 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:48:46.356850 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb091c5_dc9e_4c16_a492_c94bce01faca.slice/crio-e8c1d5928996ce08d10e7f233e323625bce99bbf2d06c12b4556cafd997063e0 WatchSource:0}: Error finding container e8c1d5928996ce08d10e7f233e323625bce99bbf2d06c12b4556cafd997063e0: Status 404 returned error can't find the container with id e8c1d5928996ce08d10e7f233e323625bce99bbf2d06c12b4556cafd997063e0 Sep 15 20:48:46.670254 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.670182 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45"] Sep 15 20:48:46.676238 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.676214 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45"] Sep 15 20:48:46.693068 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.693027 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt"] Sep 15 20:48:46.697928 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.697907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:48:46.712055 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.712033 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt"] Sep 15 20:48:46.747134 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.747106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85s8n\" (UniqueName: \"kubernetes.io/projected/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-kube-api-access-85s8n\") pod \"kuadrant-operator-controller-manager-5f685c496f-m6tvt\" (UID: \"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:48:46.747296 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.747271 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-m6tvt\" (UID: \"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:48:46.747679 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.747659 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk"] Sep 15 20:48:46.747997 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.747927 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" podUID="7abf7291-c178-4afe-95da-d3e132228be0" containerName="manager" containerID="cri-o://3e2fe2734a3939470b72462e78a85d6917740fa04c9a92e6b3e48c1ff890b56c" gracePeriod=2 Sep 15 20:48:46.755533 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.755508 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk"] Sep 15 20:48:46.762760 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:48:46.762723 2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"7abf7291-c178-4afe-95da-d3e132228be0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-pwxbk\": pods \"limitador-operator-controller-manager-74495c69f8-pwxbk\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:46.774158 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.774133 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d"] Sep 15 20:48:46.774604 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.774587 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abf7291-c178-4afe-95da-d3e132228be0" containerName="manager" Sep 15 20:48:46.774680 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.774606 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abf7291-c178-4afe-95da-d3e132228be0" containerName="manager" Sep 15 20:48:46.774733 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.774708 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abf7291-c178-4afe-95da-d3e132228be0" containerName="manager" Sep 15 20:48:46.777568 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.777550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d" Sep 15 20:48:46.779469 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:48:46.779442 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-pwxbk\" is forbidden: User \"system:node:ip-10-0-130-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-107.ec2.internal' and this object" podUID="7abf7291-c178-4afe-95da-d3e132228be0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:46.789905 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.789883 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d"] Sep 15 20:48:46.847949 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.847923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85s8n\" (UniqueName: \"kubernetes.io/projected/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-kube-api-access-85s8n\") pod \"kuadrant-operator-controller-manager-5f685c496f-m6tvt\" (UID: \"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:48:46.848063 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.848002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6xt\" (UniqueName: \"kubernetes.io/projected/9fb4446e-1739-4825-bfb5-04c425e6cb44-kube-api-access-pq6xt\") pod \"limitador-operator-controller-manager-74495c69f8-zc59d\" (UID: \"9fb4446e-1739-4825-bfb5-04c425e6cb44\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d" Sep 15 20:48:46.848063 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.848028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-m6tvt\" (UID: \"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:48:46.848353 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.848336 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-m6tvt\" (UID: \"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:48:46.856163 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.856143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85s8n\" (UniqueName: \"kubernetes.io/projected/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-kube-api-access-85s8n\") pod \"kuadrant-operator-controller-manager-5f685c496f-m6tvt\" (UID: \"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:48:46.949133 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.949061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6xt\" (UniqueName: \"kubernetes.io/projected/9fb4446e-1739-4825-bfb5-04c425e6cb44-kube-api-access-pq6xt\") pod \"limitador-operator-controller-manager-74495c69f8-zc59d\" (UID: \"9fb4446e-1739-4825-bfb5-04c425e6cb44\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d" Sep 15 20:48:46.958321 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.958292 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6xt\" (UniqueName: \"kubernetes.io/projected/9fb4446e-1739-4825-bfb5-04c425e6cb44-kube-api-access-pq6xt\") pod \"limitador-operator-controller-manager-74495c69f8-zc59d\" (UID: \"9fb4446e-1739-4825-bfb5-04c425e6cb44\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d" Sep 15 20:48:46.962395 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.962367 2579 generic.go:356] "Generic (PLEG): container finished" podID="7abf7291-c178-4afe-95da-d3e132228be0" containerID="3e2fe2734a3939470b72462e78a85d6917740fa04c9a92e6b3e48c1ff890b56c" exitCode=0 Sep 15 20:48:46.991695 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:46.991670 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:46.993491 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:48:46.993466 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-pwxbk\" is forbidden: User \"system:node:ip-10-0-130-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-107.ec2.internal' and this object" podUID="7abf7291-c178-4afe-95da-d3e132228be0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:47.007782 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.007761 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:48:47.049816 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.049783 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dv4f\" (UniqueName: \"kubernetes.io/projected/7abf7291-c178-4afe-95da-d3e132228be0-kube-api-access-9dv4f\") pod \"7abf7291-c178-4afe-95da-d3e132228be0\" (UID: \"7abf7291-c178-4afe-95da-d3e132228be0\") " Sep 15 20:48:47.052191 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.052155 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abf7291-c178-4afe-95da-d3e132228be0-kube-api-access-9dv4f" (OuterVolumeSpecName: "kube-api-access-9dv4f") pod "7abf7291-c178-4afe-95da-d3e132228be0" (UID: "7abf7291-c178-4afe-95da-d3e132228be0"). InnerVolumeSpecName "kube-api-access-9dv4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:48:47.140036 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.139985 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d" Sep 15 20:48:47.151423 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.151389 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dv4f\" (UniqueName: \"kubernetes.io/projected/7abf7291-c178-4afe-95da-d3e132228be0-kube-api-access-9dv4f\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:48:47.286613 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.286585 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt"] Sep 15 20:48:47.290453 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:48:47.290419 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cb8fa0_45eb_47c8_afdb_f28ca5aad16d.slice/crio-a36d85b5251223a3edc913b29e7171eb88bb026e6cb2b1c8901c24e199c4b1ab WatchSource:0}: Error finding container a36d85b5251223a3edc913b29e7171eb88bb026e6cb2b1c8901c24e199c4b1ab: Status 404 returned error can't find the container with id a36d85b5251223a3edc913b29e7171eb88bb026e6cb2b1c8901c24e199c4b1ab Sep 15 20:48:47.312924 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.312889 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abf7291-c178-4afe-95da-d3e132228be0" path="/var/lib/kubelet/pods/7abf7291-c178-4afe-95da-d3e132228be0/volumes" Sep 15 20:48:47.428849 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.428820 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d"] Sep 15 20:48:47.431820 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:48:47.431794 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb4446e_1739_4825_bfb5_04c425e6cb44.slice/crio-a3228113020cbaf9daf0a74d44ec1140fab5a63ae7fa2edb0923d10eafd13769 WatchSource:0}: Error finding container a3228113020cbaf9daf0a74d44ec1140fab5a63ae7fa2edb0923d10eafd13769: Status 404 returned error can't find the container with id a3228113020cbaf9daf0a74d44ec1140fab5a63ae7fa2edb0923d10eafd13769 Sep 15 20:48:47.974043 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.973414 2579 scope.go:117] "RemoveContainer" containerID="3e2fe2734a3939470b72462e78a85d6917740fa04c9a92e6b3e48c1ff890b56c" Sep 15 20:48:47.974043 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.973603 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pwxbk" Sep 15 20:48:47.977116 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.976697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d" event={"ID":"9fb4446e-1739-4825-bfb5-04c425e6cb44","Type":"ContainerStarted","Data":"ba8ed734ddefda2c97bd736790e663f2b2803540f50b6930f3f463f56223090f"} Sep 15 20:48:47.977116 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.976868 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d" Sep 15 20:48:47.977116 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.976944 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d" event={"ID":"9fb4446e-1739-4825-bfb5-04c425e6cb44","Type":"ContainerStarted","Data":"a3228113020cbaf9daf0a74d44ec1140fab5a63ae7fa2edb0923d10eafd13769"} Sep 15 20:48:47.979241 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:47.979213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" event={"ID":"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d","Type":"ContainerStarted","Data":"a36d85b5251223a3edc913b29e7171eb88bb026e6cb2b1c8901c24e199c4b1ab"} Sep 15 20:48:48.016304 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:48.016248 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d" podStartSLOduration=2.016228239 podStartE2EDuration="2.016228239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:48:48.015753813 +0000 UTC m=+725.209707273" watchObservedRunningTime="2026-09-15 20:48:48.016228239 +0000 UTC m=+725.210181695" Sep 15 20:48:50.999251 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:50.999216 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" podUID="6bb091c5-dc9e-4c16-a492-c94bce01faca" containerName="manager" containerID="cri-o://f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a" gracePeriod=2 Sep 15 20:48:51.001150 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.001129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" event={"ID":"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d","Type":"ContainerStarted","Data":"3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3"} Sep 15 20:48:51.001304 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.001279 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:48:51.021771 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.021724 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" podStartSLOduration=1.9048723829999998 podStartE2EDuration="5.021708109s" totalImagesPullingTime="3.116835726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:48:51.019627638 +0000 UTC m=+728.213581094" watchObservedRunningTime="2026-09-15 20:48:51.021708109 +0000 UTC m=+728.215661565" Sep 15 20:48:51.234845 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.234823 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:51.236822 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:48:51.236799 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-5zv45\" is forbidden: User \"system:node:ip-10-0-130-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-107.ec2.internal' and this object" podUID="6bb091c5-dc9e-4c16-a492-c94bce01faca" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:51.289428 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.289395 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7nrn\" (UniqueName: \"kubernetes.io/projected/6bb091c5-dc9e-4c16-a492-c94bce01faca-kube-api-access-k7nrn\") pod \"6bb091c5-dc9e-4c16-a492-c94bce01faca\" (UID: \"6bb091c5-dc9e-4c16-a492-c94bce01faca\") " Sep 15 20:48:51.289617 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.289548 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bb091c5-dc9e-4c16-a492-c94bce01faca-extensions-socket-volume\") pod \"6bb091c5-dc9e-4c16-a492-c94bce01faca\" (UID: \"6bb091c5-dc9e-4c16-a492-c94bce01faca\") " Sep 15 20:48:51.289927 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.289904 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb091c5-dc9e-4c16-a492-c94bce01faca-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6bb091c5-dc9e-4c16-a492-c94bce01faca" (UID: "6bb091c5-dc9e-4c16-a492-c94bce01faca"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:48:51.291783 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.291762 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb091c5-dc9e-4c16-a492-c94bce01faca-kube-api-access-k7nrn" (OuterVolumeSpecName: "kube-api-access-k7nrn") pod "6bb091c5-dc9e-4c16-a492-c94bce01faca" (UID: "6bb091c5-dc9e-4c16-a492-c94bce01faca"). InnerVolumeSpecName "kube-api-access-k7nrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:48:51.310522 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.310463 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb091c5-dc9e-4c16-a492-c94bce01faca" path="/var/lib/kubelet/pods/6bb091c5-dc9e-4c16-a492-c94bce01faca/volumes" Sep 15 20:48:51.390895 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.390863 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bb091c5-dc9e-4c16-a492-c94bce01faca-extensions-socket-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:48:51.390895 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:51.390887 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7nrn\" (UniqueName: \"kubernetes.io/projected/6bb091c5-dc9e-4c16-a492-c94bce01faca-kube-api-access-k7nrn\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:48:52.007400 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:52.007370 2579 generic.go:356] "Generic (PLEG): container finished" podID="6bb091c5-dc9e-4c16-a492-c94bce01faca" containerID="f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a" exitCode=2 Sep 15 20:48:52.007854 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:52.007412 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:52.007854 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:52.007480 2579 scope.go:117] "RemoveContainer" containerID="f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a" Sep 15 20:48:52.011579 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:48:52.011555 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-5zv45\" is forbidden: User \"system:node:ip-10-0-130-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-107.ec2.internal' and this object" podUID="6bb091c5-dc9e-4c16-a492-c94bce01faca" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:52.015716 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:52.015692 2579 scope.go:117] "RemoveContainer" containerID="f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a" Sep 15 20:48:52.015997 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:48:52.015977 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a\": container with ID starting with f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a not found: ID does not exist" containerID="f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a" Sep 15 20:48:52.016062 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:52.016003 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a"} err="failed to get container status \"f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a\": rpc error: code = NotFound desc = could not find container \"f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a\": container with ID starting with f51a470eacc40924604c5a258a0e89ad56028a861e841b63940e454d9933755a not found: ID does not exist" Sep 15 20:48:53.310512 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:48:53.310464 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-5zv45\" is forbidden: User \"system:node:ip-10-0-130-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-107.ec2.internal' and this object" podUID="6bb091c5-dc9e-4c16-a492-c94bce01faca" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5zv45" Sep 15 20:48:58.989422 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:48:58.989389 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zc59d" Sep 15 20:49:02.010341 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:02.010313 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:49:14.507779 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:14.507748 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt"] Sep 15 20:49:14.508237 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:14.507960 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" podUID="e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d" containerName="manager" containerID="cri-o://3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3" gracePeriod=10 Sep 15 20:49:14.755342 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:14.755313 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:49:14.889771 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:14.889733 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85s8n\" (UniqueName: \"kubernetes.io/projected/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-kube-api-access-85s8n\") pod \"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d\" (UID: \"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d\") " Sep 15 20:49:14.889771 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:14.889773 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-extensions-socket-volume\") pod \"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d\" (UID: \"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d\") " Sep 15 20:49:14.890276 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:14.890248 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d" (UID: "e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:49:14.892058 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:14.892026 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-kube-api-access-85s8n" (OuterVolumeSpecName: "kube-api-access-85s8n") pod "e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d" (UID: "e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d"). InnerVolumeSpecName "kube-api-access-85s8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:49:14.991154 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:14.991115 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85s8n\" (UniqueName: \"kubernetes.io/projected/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-kube-api-access-85s8n\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:49:14.991154 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:14.991147 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d-extensions-socket-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:49:15.115428 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:15.115398 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d" containerID="3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3" exitCode=0 Sep 15 20:49:15.115613 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:15.115445 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" Sep 15 20:49:15.115613 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:15.115473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" event={"ID":"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d","Type":"ContainerDied","Data":"3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3"} Sep 15 20:49:15.115613 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:15.115545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt" event={"ID":"e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d","Type":"ContainerDied","Data":"a36d85b5251223a3edc913b29e7171eb88bb026e6cb2b1c8901c24e199c4b1ab"} Sep 15 20:49:15.115613 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:15.115569 2579 scope.go:117] "RemoveContainer" containerID="3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3" Sep 15 20:49:15.131361 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:15.131337 2579 scope.go:117] "RemoveContainer" containerID="3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3" Sep 15 20:49:15.131729 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:49:15.131706 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3\": container with ID starting with 3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3 not found: ID does not exist" containerID="3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3" Sep 15 20:49:15.131803 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:15.131739 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3"} err="failed to get container status \"3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3\": rpc error: code = NotFound desc = could not find container \"3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3\": container with ID starting with 3d0e67db29d182da9ed079e90239d385a042a87f51a42d8dff0b872a1ce05dc3 not found: ID does not exist" Sep 15 20:49:15.143068 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:15.143007 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt"] Sep 15 20:49:15.149073 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:15.149052 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-m6tvt"] Sep 15 20:49:15.310716 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:15.310680 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d" path="/var/lib/kubelet/pods/e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d/volumes" Sep 15 20:49:31.606207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.606176 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zf6md"] Sep 15 20:49:31.606717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.606474 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb091c5-dc9e-4c16-a492-c94bce01faca" containerName="manager" Sep 15 20:49:31.606717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.606485 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb091c5-dc9e-4c16-a492-c94bce01faca" containerName="manager" Sep 15 20:49:31.606717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.606519 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d" containerName="manager" Sep 15 20:49:31.606717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.606525 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d" containerName="manager" Sep 15 20:49:31.606717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.606573 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb091c5-dc9e-4c16-a492-c94bce01faca" containerName="manager" Sep 15 20:49:31.606717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.606582 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9cb8fa0-45eb-47c8-afdb-f28ca5aad16d" containerName="manager" Sep 15 20:49:31.609896 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.609880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" Sep 15 20:49:31.612474 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.612450 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6kdkm\"" Sep 15 20:49:31.618991 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.618969 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zf6md"] Sep 15 20:49:31.627146 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.627121 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xq7\" (UniqueName: \"kubernetes.io/projected/0058f187-02b4-4544-a5fa-9cda2473b7cf-kube-api-access-k8xq7\") pod \"authorino-7bcfd8f9cd-zf6md\" (UID: \"0058f187-02b4-4544-a5fa-9cda2473b7cf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" Sep 15 20:49:31.727984 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.727954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xq7\" (UniqueName: \"kubernetes.io/projected/0058f187-02b4-4544-a5fa-9cda2473b7cf-kube-api-access-k8xq7\") pod \"authorino-7bcfd8f9cd-zf6md\" (UID: \"0058f187-02b4-4544-a5fa-9cda2473b7cf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" Sep 15 20:49:31.735994 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.735964 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xq7\" (UniqueName: \"kubernetes.io/projected/0058f187-02b4-4544-a5fa-9cda2473b7cf-kube-api-access-k8xq7\") pod \"authorino-7bcfd8f9cd-zf6md\" (UID: \"0058f187-02b4-4544-a5fa-9cda2473b7cf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" Sep 15 20:49:31.919024 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:31.918927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" Sep 15 20:49:32.185291 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:32.185159 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zf6md"] Sep 15 20:49:32.187883 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:49:32.187857 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0058f187_02b4_4544_a5fa_9cda2473b7cf.slice/crio-528caf296b7a567d6183bfed3ca1a8785a9ae86cc6d5fbcb0451fedc3ebdd36f WatchSource:0}: Error finding container 528caf296b7a567d6183bfed3ca1a8785a9ae86cc6d5fbcb0451fedc3ebdd36f: Status 404 returned error can't find the container with id 528caf296b7a567d6183bfed3ca1a8785a9ae86cc6d5fbcb0451fedc3ebdd36f Sep 15 20:49:32.199661 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:32.199627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" event={"ID":"0058f187-02b4-4544-a5fa-9cda2473b7cf","Type":"ContainerStarted","Data":"528caf296b7a567d6183bfed3ca1a8785a9ae86cc6d5fbcb0451fedc3ebdd36f"} Sep 15 20:49:34.209861 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:34.209823 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" event={"ID":"0058f187-02b4-4544-a5fa-9cda2473b7cf","Type":"ContainerStarted","Data":"9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239"} Sep 15 20:49:34.224280 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:34.224233 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" podStartSLOduration=1.366902991 podStartE2EDuration="3.224218527s" totalImagesPullingTime="1.857315536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:49:34.223652624 +0000 UTC m=+771.417606089" watchObservedRunningTime="2026-09-15 20:49:34.224218527 +0000 UTC m=+771.418172072" Sep 15 20:49:34.665702 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:34.665668 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zf6md"] Sep 15 20:49:36.219043 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:36.219002 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" podUID="0058f187-02b4-4544-a5fa-9cda2473b7cf" containerName="authorino" containerID="cri-o://9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239" gracePeriod=30 Sep 15 20:49:36.461238 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:36.461213 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" Sep 15 20:49:36.571136 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:36.571111 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8xq7\" (UniqueName: \"kubernetes.io/projected/0058f187-02b4-4544-a5fa-9cda2473b7cf-kube-api-access-k8xq7\") pod \"0058f187-02b4-4544-a5fa-9cda2473b7cf\" (UID: \"0058f187-02b4-4544-a5fa-9cda2473b7cf\") " Sep 15 20:49:36.573302 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:36.573264 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0058f187-02b4-4544-a5fa-9cda2473b7cf-kube-api-access-k8xq7" (OuterVolumeSpecName: "kube-api-access-k8xq7") pod "0058f187-02b4-4544-a5fa-9cda2473b7cf" (UID: "0058f187-02b4-4544-a5fa-9cda2473b7cf"). InnerVolumeSpecName "kube-api-access-k8xq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:49:36.671659 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:36.671623 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8xq7\" (UniqueName: \"kubernetes.io/projected/0058f187-02b4-4544-a5fa-9cda2473b7cf-kube-api-access-k8xq7\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:49:37.224151 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:37.224109 2579 generic.go:356] "Generic (PLEG): container finished" podID="0058f187-02b4-4544-a5fa-9cda2473b7cf" containerID="9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239" exitCode=0 Sep 15 20:49:37.224151 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:37.224151 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" Sep 15 20:49:37.224717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:37.224158 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" event={"ID":"0058f187-02b4-4544-a5fa-9cda2473b7cf","Type":"ContainerDied","Data":"9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239"} Sep 15 20:49:37.224717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:37.224203 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zf6md" event={"ID":"0058f187-02b4-4544-a5fa-9cda2473b7cf","Type":"ContainerDied","Data":"528caf296b7a567d6183bfed3ca1a8785a9ae86cc6d5fbcb0451fedc3ebdd36f"} Sep 15 20:49:37.224717 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:37.224221 2579 scope.go:117] "RemoveContainer" containerID="9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239" Sep 15 20:49:37.233032 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:37.233018 2579 scope.go:117] "RemoveContainer" containerID="9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239" Sep 15 20:49:37.233303 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:49:37.233283 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239\": container with ID starting with 9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239 not found: ID does not exist" containerID="9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239" Sep 15 20:49:37.233371 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:37.233308 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239"} err="failed to get container status \"9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239\": rpc error: code = NotFound desc = could not find container \"9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239\": container with ID starting with 9fff9c5fa71deb872aa01b00d2286fd955f6c5c5e3979ac1feb188a68cf84239 not found: ID does not exist" Sep 15 20:49:37.243206 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:37.243183 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zf6md"] Sep 15 20:49:37.244951 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:37.244929 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zf6md"] Sep 15 20:49:37.310720 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:37.310686 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0058f187-02b4-4544-a5fa-9cda2473b7cf" path="/var/lib/kubelet/pods/0058f187-02b4-4544-a5fa-9cda2473b7cf/volumes" Sep 15 20:49:47.664370 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.664331 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s"] Sep 15 20:49:47.664791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.664670 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0058f187-02b4-4544-a5fa-9cda2473b7cf" containerName="authorino" Sep 15 20:49:47.664791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.664682 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0058f187-02b4-4544-a5fa-9cda2473b7cf" containerName="authorino" Sep 15 20:49:47.664791 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.664742 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0058f187-02b4-4544-a5fa-9cda2473b7cf" containerName="authorino" Sep 15 20:49:47.669659 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.669639 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" Sep 15 20:49:47.672352 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.672108 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 20:49:47.672352 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.672183 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 20:49:47.672352 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.672183 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-n9c7p\"" Sep 15 20:49:47.672352 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.672189 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 15 20:49:47.674281 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.674258 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s"] Sep 15 20:49:47.769467 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.769421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eaf70320-ef57-4195-9f6f-fb7da3b9575a-data\") pod \"postgres-69fb4594cf-xhn8s\" (UID: \"eaf70320-ef57-4195-9f6f-fb7da3b9575a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" Sep 15 20:49:47.769467 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.769463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twlk\" (UniqueName: \"kubernetes.io/projected/eaf70320-ef57-4195-9f6f-fb7da3b9575a-kube-api-access-9twlk\") pod \"postgres-69fb4594cf-xhn8s\" (UID: \"eaf70320-ef57-4195-9f6f-fb7da3b9575a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" Sep 15 20:49:47.870829 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.870790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eaf70320-ef57-4195-9f6f-fb7da3b9575a-data\") pod \"postgres-69fb4594cf-xhn8s\" (UID: \"eaf70320-ef57-4195-9f6f-fb7da3b9575a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" Sep 15 20:49:47.870829 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.870827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twlk\" (UniqueName: \"kubernetes.io/projected/eaf70320-ef57-4195-9f6f-fb7da3b9575a-kube-api-access-9twlk\") pod \"postgres-69fb4594cf-xhn8s\" (UID: \"eaf70320-ef57-4195-9f6f-fb7da3b9575a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" Sep 15 20:49:47.871177 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.871157 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eaf70320-ef57-4195-9f6f-fb7da3b9575a-data\") pod \"postgres-69fb4594cf-xhn8s\" (UID: \"eaf70320-ef57-4195-9f6f-fb7da3b9575a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" Sep 15 20:49:47.879138 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.879106 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twlk\" (UniqueName: \"kubernetes.io/projected/eaf70320-ef57-4195-9f6f-fb7da3b9575a-kube-api-access-9twlk\") pod \"postgres-69fb4594cf-xhn8s\" (UID: \"eaf70320-ef57-4195-9f6f-fb7da3b9575a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" Sep 15 20:49:47.981064 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:47.980989 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" Sep 15 20:49:48.249642 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:48.249564 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s"] Sep 15 20:49:48.253237 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:49:48.253209 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf70320_ef57_4195_9f6f_fb7da3b9575a.slice/crio-1b98845dc888d83a3d37dd8156531461077807f50d506e0b7e5e3079687cf726 WatchSource:0}: Error finding container 1b98845dc888d83a3d37dd8156531461077807f50d506e0b7e5e3079687cf726: Status 404 returned error can't find the container with id 1b98845dc888d83a3d37dd8156531461077807f50d506e0b7e5e3079687cf726 Sep 15 20:49:48.274776 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:48.274745 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" event={"ID":"eaf70320-ef57-4195-9f6f-fb7da3b9575a","Type":"ContainerStarted","Data":"1b98845dc888d83a3d37dd8156531461077807f50d506e0b7e5e3079687cf726"} Sep 15 20:49:54.319308 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:54.319276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" event={"ID":"eaf70320-ef57-4195-9f6f-fb7da3b9575a","Type":"ContainerStarted","Data":"3c977741f5c18c937a95184a750520f28be645580f1c7f7bf32c43eb2c32e2a9"} Sep 15 20:49:54.319707 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:54.319414 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" Sep 15 20:49:54.335024 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:49:54.334980 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" podStartSLOduration=2.323671982 podStartE2EDuration="7.334962666s" totalImagesPullingTime="5.011290684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:49:54.333653624 +0000 UTC m=+791.527607079" watchObservedRunningTime="2026-09-15 20:49:54.334962666 +0000 UTC m=+791.528916121" Sep 15 20:50:00.352414 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:00.352377 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xhn8s" Sep 15 20:50:00.923076 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:00.923042 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-dk8tw"] Sep 15 20:50:00.928067 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:00.928049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dk8tw" Sep 15 20:50:00.930401 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:00.930380 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6kdkm\"" Sep 15 20:50:00.935455 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:00.935435 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-dk8tw"] Sep 15 20:50:00.960668 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:00.960639 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dk8tw"] Sep 15 20:50:00.960883 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:50:00.960865 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lvcz6], unattached volumes=[], failed to process volumes=[kube-api-access-lvcz6]: context canceled" pod="kuadrant-system/authorino-557f876f69-dk8tw" podUID="8c4fc2f2-339e-4198-b27e-580bd5a5c948" Sep 15 20:50:00.986344 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:00.986309 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5tqsj"] Sep 15 20:50:00.989895 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:00.989875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" Sep 15 20:50:00.992186 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:00.992165 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 15 20:50:00.995410 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:00.995380 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5tqsj"] Sep 15 20:50:01.086399 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.086358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvcz6\" (UniqueName: \"kubernetes.io/projected/8c4fc2f2-339e-4198-b27e-580bd5a5c948-kube-api-access-lvcz6\") pod \"authorino-557f876f69-dk8tw\" (UID: \"8c4fc2f2-339e-4198-b27e-580bd5a5c948\") " pod="kuadrant-system/authorino-557f876f69-dk8tw" Sep 15 20:50:01.086399 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.086398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66dad37d-ba66-4d0b-a1e3-bef3509a9874-tls-cert\") pod \"authorino-6d4dfc649f-5tqsj\" (UID: \"66dad37d-ba66-4d0b-a1e3-bef3509a9874\") " pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" Sep 15 20:50:01.086677 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.086528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmkn\" (UniqueName: \"kubernetes.io/projected/66dad37d-ba66-4d0b-a1e3-bef3509a9874-kube-api-access-qvmkn\") pod \"authorino-6d4dfc649f-5tqsj\" (UID: \"66dad37d-ba66-4d0b-a1e3-bef3509a9874\") " pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" Sep 15 20:50:01.168170 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.168130 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5tqsj"] Sep 15 20:50:01.168430 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:50:01.168407 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qvmkn tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" podUID="66dad37d-ba66-4d0b-a1e3-bef3509a9874" Sep 15 20:50:01.187888 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.187820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66dad37d-ba66-4d0b-a1e3-bef3509a9874-tls-cert\") pod \"authorino-6d4dfc649f-5tqsj\" (UID: \"66dad37d-ba66-4d0b-a1e3-bef3509a9874\") " pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" Sep 15 20:50:01.187992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.187888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmkn\" (UniqueName: \"kubernetes.io/projected/66dad37d-ba66-4d0b-a1e3-bef3509a9874-kube-api-access-qvmkn\") pod \"authorino-6d4dfc649f-5tqsj\" (UID: \"66dad37d-ba66-4d0b-a1e3-bef3509a9874\") " pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" Sep 15 20:50:01.187992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.187931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvcz6\" (UniqueName: \"kubernetes.io/projected/8c4fc2f2-339e-4198-b27e-580bd5a5c948-kube-api-access-lvcz6\") pod \"authorino-557f876f69-dk8tw\" (UID: \"8c4fc2f2-339e-4198-b27e-580bd5a5c948\") " pod="kuadrant-system/authorino-557f876f69-dk8tw" Sep 15 20:50:01.190446 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.190422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66dad37d-ba66-4d0b-a1e3-bef3509a9874-tls-cert\") pod \"authorino-6d4dfc649f-5tqsj\" (UID: \"66dad37d-ba66-4d0b-a1e3-bef3509a9874\") " pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" Sep 15 20:50:01.198216 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.198186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmkn\" (UniqueName: \"kubernetes.io/projected/66dad37d-ba66-4d0b-a1e3-bef3509a9874-kube-api-access-qvmkn\") pod \"authorino-6d4dfc649f-5tqsj\" (UID: \"66dad37d-ba66-4d0b-a1e3-bef3509a9874\") " pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" Sep 15 20:50:01.198441 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.198422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvcz6\" (UniqueName: \"kubernetes.io/projected/8c4fc2f2-339e-4198-b27e-580bd5a5c948-kube-api-access-lvcz6\") pod \"authorino-557f876f69-dk8tw\" (UID: \"8c4fc2f2-339e-4198-b27e-580bd5a5c948\") " pod="kuadrant-system/authorino-557f876f69-dk8tw" Sep 15 20:50:01.347672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.347636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dk8tw" Sep 15 20:50:01.347838 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.347640 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" Sep 15 20:50:01.355631 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.355608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dk8tw" Sep 15 20:50:01.358915 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.358889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" Sep 15 20:50:01.491068 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.490981 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvmkn\" (UniqueName: \"kubernetes.io/projected/66dad37d-ba66-4d0b-a1e3-bef3509a9874-kube-api-access-qvmkn\") pod \"66dad37d-ba66-4d0b-a1e3-bef3509a9874\" (UID: \"66dad37d-ba66-4d0b-a1e3-bef3509a9874\") " Sep 15 20:50:01.491068 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.491036 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66dad37d-ba66-4d0b-a1e3-bef3509a9874-tls-cert\") pod \"66dad37d-ba66-4d0b-a1e3-bef3509a9874\" (UID: \"66dad37d-ba66-4d0b-a1e3-bef3509a9874\") " Sep 15 20:50:01.491277 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.491139 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvcz6\" (UniqueName: \"kubernetes.io/projected/8c4fc2f2-339e-4198-b27e-580bd5a5c948-kube-api-access-lvcz6\") pod \"8c4fc2f2-339e-4198-b27e-580bd5a5c948\" (UID: \"8c4fc2f2-339e-4198-b27e-580bd5a5c948\") " Sep 15 20:50:01.493237 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.493205 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66dad37d-ba66-4d0b-a1e3-bef3509a9874-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "66dad37d-ba66-4d0b-a1e3-bef3509a9874" (UID: "66dad37d-ba66-4d0b-a1e3-bef3509a9874"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:50:01.493237 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.493229 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66dad37d-ba66-4d0b-a1e3-bef3509a9874-kube-api-access-qvmkn" (OuterVolumeSpecName: "kube-api-access-qvmkn") pod "66dad37d-ba66-4d0b-a1e3-bef3509a9874" (UID: "66dad37d-ba66-4d0b-a1e3-bef3509a9874"). InnerVolumeSpecName "kube-api-access-qvmkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:50:01.493396 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.493208 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4fc2f2-339e-4198-b27e-580bd5a5c948-kube-api-access-lvcz6" (OuterVolumeSpecName: "kube-api-access-lvcz6") pod "8c4fc2f2-339e-4198-b27e-580bd5a5c948" (UID: "8c4fc2f2-339e-4198-b27e-580bd5a5c948"). InnerVolumeSpecName "kube-api-access-lvcz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:50:01.591954 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.591911 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmkn\" (UniqueName: \"kubernetes.io/projected/66dad37d-ba66-4d0b-a1e3-bef3509a9874-kube-api-access-qvmkn\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:50:01.591954 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.591946 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66dad37d-ba66-4d0b-a1e3-bef3509a9874-tls-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:50:01.591954 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:01.591958 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvcz6\" (UniqueName: \"kubernetes.io/projected/8c4fc2f2-339e-4198-b27e-580bd5a5c948-kube-api-access-lvcz6\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:50:02.351646 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:02.351616 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dk8tw" Sep 15 20:50:02.351824 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:02.351623 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5tqsj" Sep 15 20:50:02.384419 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:02.384390 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dk8tw"] Sep 15 20:50:02.387203 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:02.387179 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-dk8tw"] Sep 15 20:50:02.414596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:02.414562 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5tqsj"] Sep 15 20:50:02.420491 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:02.420463 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5tqsj"] Sep 15 20:50:03.311430 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:03.311392 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66dad37d-ba66-4d0b-a1e3-bef3509a9874" path="/var/lib/kubelet/pods/66dad37d-ba66-4d0b-a1e3-bef3509a9874/volumes" Sep 15 20:50:03.311663 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:50:03.311649 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c4fc2f2-339e-4198-b27e-580bd5a5c948" path="/var/lib/kubelet/pods/8c4fc2f2-339e-4198-b27e-580bd5a5c948/volumes" Sep 15 20:51:00.266386 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.266352 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn"] Sep 15 20:51:00.270161 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.270144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.272350 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.272326 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bwfmg\"" Sep 15 20:51:00.272465 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.272326 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 15 20:51:00.272465 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.272333 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 15 20:51:00.273000 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.272977 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 15 20:51:00.278588 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.278565 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn"] Sep 15 20:51:00.365264 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.365225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1396b69c-8226-49b6-a226-e71568470df2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.365437 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.365291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.365437 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.365310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.365437 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.365399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77hng\" (UniqueName: \"kubernetes.io/projected/1396b69c-8226-49b6-a226-e71568470df2-kube-api-access-77hng\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.365596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.365435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.365596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.365454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.466204 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.466171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.466204 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.466206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.466459 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.466260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77hng\" (UniqueName: \"kubernetes.io/projected/1396b69c-8226-49b6-a226-e71568470df2-kube-api-access-77hng\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.466459 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.466285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.466459 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.466306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.466459 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.466339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1396b69c-8226-49b6-a226-e71568470df2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.466699 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.466676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.466777 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.466741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.466852 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.466779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.468704 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.468666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1396b69c-8226-49b6-a226-e71568470df2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.468860 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.468845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1396b69c-8226-49b6-a226-e71568470df2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.475787 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.475763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77hng\" (UniqueName: \"kubernetes.io/projected/1396b69c-8226-49b6-a226-e71568470df2-kube-api-access-77hng\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn\" (UID: \"1396b69c-8226-49b6-a226-e71568470df2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.581690 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.581613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:00.853259 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:00.853229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn"] Sep 15 20:51:00.855850 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:51:00.855823 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1396b69c_8226_49b6_a226_e71568470df2.slice/crio-0e2f92f5c439a8ba6a5047c14f08ef1aa76ade0bfae977b289ffcd1d461acacc WatchSource:0}: Error finding container 0e2f92f5c439a8ba6a5047c14f08ef1aa76ade0bfae977b289ffcd1d461acacc: Status 404 returned error can't find the container with id 0e2f92f5c439a8ba6a5047c14f08ef1aa76ade0bfae977b289ffcd1d461acacc Sep 15 20:51:01.609601 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:01.609558 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" event={"ID":"1396b69c-8226-49b6-a226-e71568470df2","Type":"ContainerStarted","Data":"0e2f92f5c439a8ba6a5047c14f08ef1aa76ade0bfae977b289ffcd1d461acacc"} Sep 15 20:51:02.615068 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:02.615036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" event={"ID":"1396b69c-8226-49b6-a226-e71568470df2","Type":"ContainerStarted","Data":"972c5f1afb444be1b833ba6ecb39e623138ea2fc7baaea11f3d399dd72524b09"} Sep 15 20:51:02.634771 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:02.634718 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" podStartSLOduration=1.619894284 podStartE2EDuration="2.634702118s" totalImagesPullingTime="1.014807834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:51:02.633382481 +0000 UTC m=+859.827335937" watchObservedRunningTime="2026-09-15 20:51:02.634702118 +0000 UTC m=+859.828655572" Sep 15 20:51:10.582488 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:10.582454 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:10.582963 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:10.582556 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:10.594976 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:10.594949 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:10.667094 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:10.667062 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn" Sep 15 20:51:53.062985 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.062953 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng"] Sep 15 20:51:53.066431 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.066416 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.068756 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.068735 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 15 20:51:53.076438 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.076416 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng"] Sep 15 20:51:53.196363 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.196331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwg6d\" (UniqueName: \"kubernetes.io/projected/f53f7be1-d8a4-43fd-8d12-95396993655b-kube-api-access-kwg6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.196561 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.196397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.196561 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.196417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.196561 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.196445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.196561 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.196474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.196561 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.196488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53f7be1-d8a4-43fd-8d12-95396993655b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.297560 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.297525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.297560 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.297564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.297781 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.297590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.297781 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.297626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.297781 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.297647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53f7be1-d8a4-43fd-8d12-95396993655b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.297781 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.297691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwg6d\" (UniqueName: \"kubernetes.io/projected/f53f7be1-d8a4-43fd-8d12-95396993655b-kube-api-access-kwg6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.297971 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.297930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.297971 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.297960 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.298091 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.298070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.299965 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.299944 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53f7be1-d8a4-43fd-8d12-95396993655b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.300266 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.300248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53f7be1-d8a4-43fd-8d12-95396993655b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.305065 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.305039 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwg6d\" (UniqueName: \"kubernetes.io/projected/f53f7be1-d8a4-43fd-8d12-95396993655b-kube-api-access-kwg6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n8hng\" (UID: \"f53f7be1-d8a4-43fd-8d12-95396993655b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.376715 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.376617 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:51:53.646833 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.646802 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng"] Sep 15 20:51:53.649652 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:51:53.649618 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53f7be1_d8a4_43fd_8d12_95396993655b.slice/crio-fcebc4ff625d527d662c1e2be0b037774e36c735de119bbf948e94de8a0e1810 WatchSource:0}: Error finding container fcebc4ff625d527d662c1e2be0b037774e36c735de119bbf948e94de8a0e1810: Status 404 returned error can't find the container with id fcebc4ff625d527d662c1e2be0b037774e36c735de119bbf948e94de8a0e1810 Sep 15 20:51:53.651344 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.651326 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 20:51:53.841422 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:53.841392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" event={"ID":"f53f7be1-d8a4-43fd-8d12-95396993655b","Type":"ContainerStarted","Data":"fcebc4ff625d527d662c1e2be0b037774e36c735de119bbf948e94de8a0e1810"} Sep 15 20:51:54.847080 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:54.847039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" event={"ID":"f53f7be1-d8a4-43fd-8d12-95396993655b","Type":"ContainerStarted","Data":"541f4cdde3f4f259f33f897c147911c2dd0afbb0c2a465bae41c9222e7208e1e"} Sep 15 20:51:54.862182 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:51:54.862132 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" podStartSLOduration=1.671820888 podStartE2EDuration="1.86211488s" totalImagesPullingTime="190.293992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:51:54.861528736 +0000 UTC m=+912.055482188" watchObservedRunningTime="2026-09-15 20:51:54.86211488 +0000 UTC m=+912.056068335" Sep 15 20:52:03.377211 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:03.377168 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:52:03.377723 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:03.377242 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:52:03.389639 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:03.389612 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:52:03.896697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:03.896670 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n8hng" Sep 15 20:52:03.962409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:03.962377 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb"] Sep 15 20:52:03.965778 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:03.965759 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:03.967999 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:03.967976 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 15 20:52:03.978324 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:03.978299 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb"] Sep 15 20:52:04.093782 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.093753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.094016 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.093796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.094016 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.093888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.094016 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.093921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rb2\" (UniqueName: \"kubernetes.io/projected/04987fbe-82b1-4e39-b32c-2144082fb22d-kube-api-access-82rb2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.094016 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.093942 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04987fbe-82b1-4e39-b32c-2144082fb22d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.094016 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.094016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.195290 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.195196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.195290 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.195258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.195537 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.195291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.195537 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.195344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.195537 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.195373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rb2\" (UniqueName: \"kubernetes.io/projected/04987fbe-82b1-4e39-b32c-2144082fb22d-kube-api-access-82rb2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.195537 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.195398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04987fbe-82b1-4e39-b32c-2144082fb22d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.195751 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.195679 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.195808 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.195747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.195856 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.195840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.197746 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.197718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04987fbe-82b1-4e39-b32c-2144082fb22d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.197977 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.197962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04987fbe-82b1-4e39-b32c-2144082fb22d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.203143 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.203121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rb2\" (UniqueName: \"kubernetes.io/projected/04987fbe-82b1-4e39-b32c-2144082fb22d-kube-api-access-82rb2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb\" (UID: \"04987fbe-82b1-4e39-b32c-2144082fb22d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.275841 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.275808 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:04.543850 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.543816 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb"] Sep 15 20:52:04.547133 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:52:04.547098 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04987fbe_82b1_4e39_b32c_2144082fb22d.slice/crio-8096514b3a5f6e96cbeaabdda989389901693624d06203fc9a246d68105d77d8 WatchSource:0}: Error finding container 8096514b3a5f6e96cbeaabdda989389901693624d06203fc9a246d68105d77d8: Status 404 returned error can't find the container with id 8096514b3a5f6e96cbeaabdda989389901693624d06203fc9a246d68105d77d8 Sep 15 20:52:04.891436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.891342 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" event={"ID":"04987fbe-82b1-4e39-b32c-2144082fb22d","Type":"ContainerStarted","Data":"9e587c90effba74811f5ed708e5124e516ce3a5be8375ca09e8a91fa6935d8e7"} Sep 15 20:52:04.891436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.891390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" event={"ID":"04987fbe-82b1-4e39-b32c-2144082fb22d","Type":"ContainerStarted","Data":"8096514b3a5f6e96cbeaabdda989389901693624d06203fc9a246d68105d77d8"} Sep 15 20:52:04.910099 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:04.910050 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" podStartSLOduration=1.720864689 podStartE2EDuration="1.910033644s" totalImagesPullingTime="189.168955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:52:04.909083754 +0000 UTC m=+922.103037209" watchObservedRunningTime="2026-09-15 20:52:04.910033644 +0000 UTC m=+922.103987099" Sep 15 20:52:14.276569 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:14.276521 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:14.277077 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:14.276584 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:14.289097 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:14.289066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:52:14.946911 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:52:14.946886 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb" Sep 15 20:53:47.705703 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.705668 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74fdb965b6-f4b85"] Sep 15 20:53:47.708950 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.708932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74fdb965b6-f4b85" Sep 15 20:53:47.711628 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.711603 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 15 20:53:47.711766 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.711634 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6kdkm\"" Sep 15 20:53:47.720736 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.720714 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74fdb965b6-f4b85"] Sep 15 20:53:47.772052 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.772021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f4f92f2-3282-49e0-8f0b-ae7928c240b9-tls-cert\") pod \"authorino-74fdb965b6-f4b85\" (UID: \"2f4f92f2-3282-49e0-8f0b-ae7928c240b9\") " pod="kuadrant-system/authorino-74fdb965b6-f4b85" Sep 15 20:53:47.772234 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.772063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdpj7\" (UniqueName: \"kubernetes.io/projected/2f4f92f2-3282-49e0-8f0b-ae7928c240b9-kube-api-access-gdpj7\") pod \"authorino-74fdb965b6-f4b85\" (UID: \"2f4f92f2-3282-49e0-8f0b-ae7928c240b9\") " pod="kuadrant-system/authorino-74fdb965b6-f4b85" Sep 15 20:53:47.873007 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.872971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f4f92f2-3282-49e0-8f0b-ae7928c240b9-tls-cert\") pod \"authorino-74fdb965b6-f4b85\" (UID: \"2f4f92f2-3282-49e0-8f0b-ae7928c240b9\") " pod="kuadrant-system/authorino-74fdb965b6-f4b85" Sep 15 20:53:47.873007 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.873017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdpj7\" (UniqueName: \"kubernetes.io/projected/2f4f92f2-3282-49e0-8f0b-ae7928c240b9-kube-api-access-gdpj7\") pod \"authorino-74fdb965b6-f4b85\" (UID: \"2f4f92f2-3282-49e0-8f0b-ae7928c240b9\") " pod="kuadrant-system/authorino-74fdb965b6-f4b85" Sep 15 20:53:47.875645 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.875619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f4f92f2-3282-49e0-8f0b-ae7928c240b9-tls-cert\") pod \"authorino-74fdb965b6-f4b85\" (UID: \"2f4f92f2-3282-49e0-8f0b-ae7928c240b9\") " pod="kuadrant-system/authorino-74fdb965b6-f4b85" Sep 15 20:53:47.880587 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:47.880559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdpj7\" (UniqueName: \"kubernetes.io/projected/2f4f92f2-3282-49e0-8f0b-ae7928c240b9-kube-api-access-gdpj7\") pod \"authorino-74fdb965b6-f4b85\" (UID: \"2f4f92f2-3282-49e0-8f0b-ae7928c240b9\") " pod="kuadrant-system/authorino-74fdb965b6-f4b85" Sep 15 20:53:48.018017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:48.017988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74fdb965b6-f4b85" Sep 15 20:53:48.285206 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:48.285170 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74fdb965b6-f4b85"] Sep 15 20:53:48.287936 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:53:48.287909 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f4f92f2_3282_49e0_8f0b_ae7928c240b9.slice/crio-f3f5d7758d3c84ff4e7af12d8ee1633906fe47fcf8010c5201bce8df38f3e777 WatchSource:0}: Error finding container f3f5d7758d3c84ff4e7af12d8ee1633906fe47fcf8010c5201bce8df38f3e777: Status 404 returned error can't find the container with id f3f5d7758d3c84ff4e7af12d8ee1633906fe47fcf8010c5201bce8df38f3e777 Sep 15 20:53:48.351359 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:48.351319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74fdb965b6-f4b85" event={"ID":"2f4f92f2-3282-49e0-8f0b-ae7928c240b9","Type":"ContainerStarted","Data":"f3f5d7758d3c84ff4e7af12d8ee1633906fe47fcf8010c5201bce8df38f3e777"} Sep 15 20:53:49.358113 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:49.358022 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74fdb965b6-f4b85" event={"ID":"2f4f92f2-3282-49e0-8f0b-ae7928c240b9","Type":"ContainerStarted","Data":"2280cda0c21207229f62f528accaa7194174dea6f75e09552fd941305a437c27"} Sep 15 20:53:49.373410 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:53:49.373349 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74fdb965b6-f4b85" podStartSLOduration=1.644167177 podStartE2EDuration="2.373328302s" totalImagesPullingTime="729.161125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:53:49.372394002 +0000 UTC m=+1026.566347456" watchObservedRunningTime="2026-09-15 20:53:49.373328302 +0000 UTC m=+1026.567281757" Sep 15 20:54:14.230343 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.230306 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk"] Sep 15 20:54:14.233453 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.233436 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 20:54:14.235709 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.235685 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d5b5b9-dockercfg-8vc62\"" Sep 15 20:54:14.236151 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.236129 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d5b5b9\"" Sep 15 20:54:14.243563 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.243525 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk"] Sep 15 20:54:14.306176 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.306141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llv2k\" (UniqueName: \"kubernetes.io/projected/7a9100c0-11b2-49f0-acb0-10cad430ee72-kube-api-access-llv2k\") pod \"payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk\" (UID: \"7a9100c0-11b2-49f0-acb0-10cad430ee72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 20:54:14.306176 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.306178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9100c0-11b2-49f0-acb0-10cad430ee72-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk\" (UID: \"7a9100c0-11b2-49f0-acb0-10cad430ee72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 20:54:14.406800 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.406762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llv2k\" (UniqueName: \"kubernetes.io/projected/7a9100c0-11b2-49f0-acb0-10cad430ee72-kube-api-access-llv2k\") pod \"payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk\" (UID: \"7a9100c0-11b2-49f0-acb0-10cad430ee72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 20:54:14.406993 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.406806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9100c0-11b2-49f0-acb0-10cad430ee72-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk\" (UID: \"7a9100c0-11b2-49f0-acb0-10cad430ee72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 20:54:14.407526 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.407487 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9100c0-11b2-49f0-acb0-10cad430ee72-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk\" (UID: \"7a9100c0-11b2-49f0-acb0-10cad430ee72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 20:54:14.420047 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.420023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llv2k\" (UniqueName: \"kubernetes.io/projected/7a9100c0-11b2-49f0-acb0-10cad430ee72-kube-api-access-llv2k\") pod \"payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk\" (UID: \"7a9100c0-11b2-49f0-acb0-10cad430ee72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 20:54:14.544766 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.544732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 20:54:14.813529 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:14.813487 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk"] Sep 15 20:54:14.815342 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:54:14.815312 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9100c0_11b2_49f0_acb0_10cad430ee72.slice/crio-6d5864ac5e77ebbfca40c4fc3b5d0ede54e2dd717c9b4fe6a200b9f5acea3c11 WatchSource:0}: Error finding container 6d5864ac5e77ebbfca40c4fc3b5d0ede54e2dd717c9b4fe6a200b9f5acea3c11: Status 404 returned error can't find the container with id 6d5864ac5e77ebbfca40c4fc3b5d0ede54e2dd717c9b4fe6a200b9f5acea3c11 Sep 15 20:54:15.487727 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.487690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" event={"ID":"7a9100c0-11b2-49f0-acb0-10cad430ee72","Type":"ContainerStarted","Data":"6d5864ac5e77ebbfca40c4fc3b5d0ede54e2dd717c9b4fe6a200b9f5acea3c11"} Sep 15 20:54:15.796170 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.796139 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf"] Sep 15 20:54:15.800081 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.800051 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 20:54:15.802418 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.802393 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-16bde4-dockercfg-pfnq8\"" Sep 15 20:54:15.802418 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.802408 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-16bde4\"" Sep 15 20:54:15.809565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.809189 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf"] Sep 15 20:54:15.819998 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.819970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf\" (UID: \"1ce77d63-7610-47c6-ae43-64b4eb7da4bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 20:54:15.820171 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.820138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-488wc\" (UniqueName: \"kubernetes.io/projected/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-kube-api-access-488wc\") pod \"payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf\" (UID: \"1ce77d63-7610-47c6-ae43-64b4eb7da4bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 20:54:15.921389 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.921347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-488wc\" (UniqueName: \"kubernetes.io/projected/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-kube-api-access-488wc\") pod \"payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf\" (UID: \"1ce77d63-7610-47c6-ae43-64b4eb7da4bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 20:54:15.921595 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.921431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf\" (UID: \"1ce77d63-7610-47c6-ae43-64b4eb7da4bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 20:54:15.922164 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.922144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf\" (UID: \"1ce77d63-7610-47c6-ae43-64b4eb7da4bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 20:54:15.929775 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:15.929746 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-488wc\" (UniqueName: \"kubernetes.io/projected/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-kube-api-access-488wc\") pod \"payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf\" (UID: \"1ce77d63-7610-47c6-ae43-64b4eb7da4bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 20:54:16.114323 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:16.114240 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 20:54:16.495773 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:16.495739 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" event={"ID":"7a9100c0-11b2-49f0-acb0-10cad430ee72","Type":"ContainerStarted","Data":"44bfc5b2c08d2c104f4617327b10b4374e7bf95a317b2e9f1e86b9bf1b2d22e1"} Sep 15 20:54:16.496542 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:16.495896 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 20:54:16.516264 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:16.516206 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" podStartSLOduration=0.951904515 podStartE2EDuration="2.516183316s" totalImagesPullingTime="1.564278801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:16.514680734 +0000 UTC m=+1053.708634197" watchObservedRunningTime="2026-09-15 20:54:16.516183316 +0000 UTC m=+1053.710136775" Sep 15 20:54:16.598030 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:16.598002 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf"] Sep 15 20:54:16.600152 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:54:16.600121 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce77d63_7610_47c6_ae43_64b4eb7da4bd.slice/crio-37135893602d7c9865f8e1e46a2ce818c69db1eb479528404bc2c231b1ff9e94 WatchSource:0}: Error finding container 37135893602d7c9865f8e1e46a2ce818c69db1eb479528404bc2c231b1ff9e94: Status 404 returned error can't find the container with id 37135893602d7c9865f8e1e46a2ce818c69db1eb479528404bc2c231b1ff9e94 Sep 15 20:54:17.503428 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:17.503388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" event={"ID":"1ce77d63-7610-47c6-ae43-64b4eb7da4bd","Type":"ContainerStarted","Data":"67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3"} Sep 15 20:54:17.503959 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:17.503455 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" event={"ID":"1ce77d63-7610-47c6-ae43-64b4eb7da4bd","Type":"ContainerStarted","Data":"37135893602d7c9865f8e1e46a2ce818c69db1eb479528404bc2c231b1ff9e94"} Sep 15 20:54:17.523250 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:17.523192 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" podStartSLOduration=2.5231703899999998 podStartE2EDuration="2.52317039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:17.520973771 +0000 UTC m=+1054.714927229" watchObservedRunningTime="2026-09-15 20:54:17.52317039 +0000 UTC m=+1054.717123847" Sep 15 20:54:18.507710 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:18.507680 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 20:54:22.657343 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.657312 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c"] Sep 15 20:54:22.660920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.660899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.663269 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.663243 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 15 20:54:22.663269 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.663257 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-988a09-openshift-default-dockercfg-pptdh\"" Sep 15 20:54:22.673026 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.673005 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c"] Sep 15 20:54:22.683390 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.683369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-certs\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.683492 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.683410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-credential-socket\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.683492 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.683427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-data\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.683596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.683490 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-podinfo\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.683596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.683529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istiod-ca-cert\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.683596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.683544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6gsz\" (UniqueName: \"kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-kube-api-access-t6gsz\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.683596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.683565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-envoy\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.683740 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.683628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-socket\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.683740 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.683658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-token\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.784944 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.784911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-certs\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785140 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.784981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-credential-socket\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785140 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-data\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785140 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-podinfo\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785140 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istiod-ca-cert\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785140 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6gsz\" (UniqueName: \"kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-kube-api-access-t6gsz\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785411 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-envoy\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785411 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-socket\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785411 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-token\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785602 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785477 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-data\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785602 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-credential-socket\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785788 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785767 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-socket\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.785968 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istiod-ca-cert\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.786046 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.785985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-certs\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.787751 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.787731 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-envoy\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.787944 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.787922 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-podinfo\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.794882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.794861 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-token\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.795244 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.795225 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6gsz\" (UniqueName: \"kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-kube-api-access-t6gsz\") pod \"e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:22.972807 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:22.972721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:23.256461 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:23.256436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c"] Sep 15 20:54:23.259233 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:54:23.259195 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a1a8b0_2bc8_40f8_8279_94c28f4a9877.slice/crio-8096a2e68608689b5bc3f0dfdab67faebe5f207cb33970349e8eba71e793e452 WatchSource:0}: Error finding container 8096a2e68608689b5bc3f0dfdab67faebe5f207cb33970349e8eba71e793e452: Status 404 returned error can't find the container with id 8096a2e68608689b5bc3f0dfdab67faebe5f207cb33970349e8eba71e793e452 Sep 15 20:54:23.530830 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:23.530791 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" event={"ID":"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877","Type":"ContainerStarted","Data":"8096a2e68608689b5bc3f0dfdab67faebe5f207cb33970349e8eba71e793e452"} Sep 15 20:54:23.999174 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:23.999096 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f"] Sep 15 20:54:24.003694 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.003677 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:54:24.006070 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.006043 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b9ff68\"" Sep 15 20:54:24.011836 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.011817 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f"] Sep 15 20:54:24.096079 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.096043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77q7f\" (UniqueName: \"kubernetes.io/projected/7463aa39-ce69-4a41-ba13-4fb5f8960df4-kube-api-access-77q7f\") pod \"payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f\" (UID: \"7463aa39-ce69-4a41-ba13-4fb5f8960df4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:54:24.096240 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.096150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7463aa39-ce69-4a41-ba13-4fb5f8960df4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f\" (UID: \"7463aa39-ce69-4a41-ba13-4fb5f8960df4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:54:24.197180 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.197146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7463aa39-ce69-4a41-ba13-4fb5f8960df4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f\" (UID: \"7463aa39-ce69-4a41-ba13-4fb5f8960df4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:54:24.197371 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.197253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77q7f\" (UniqueName: \"kubernetes.io/projected/7463aa39-ce69-4a41-ba13-4fb5f8960df4-kube-api-access-77q7f\") pod \"payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f\" (UID: \"7463aa39-ce69-4a41-ba13-4fb5f8960df4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:54:24.197978 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.197956 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7463aa39-ce69-4a41-ba13-4fb5f8960df4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f\" (UID: \"7463aa39-ce69-4a41-ba13-4fb5f8960df4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:54:24.206370 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.206339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77q7f\" (UniqueName: \"kubernetes.io/projected/7463aa39-ce69-4a41-ba13-4fb5f8960df4-kube-api-access-77q7f\") pod \"payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f\" (UID: \"7463aa39-ce69-4a41-ba13-4fb5f8960df4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:54:24.315088 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.315054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:54:24.592491 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:24.592459 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f"] Sep 15 20:54:24.594541 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:54:24.594491 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7463aa39_ce69_4a41_ba13_4fb5f8960df4.slice/crio-7a734c3f4e4254e5c8b110d1bf47f5e9b747c2f10616ef0fb9f66bca612fc2e3 WatchSource:0}: Error finding container 7a734c3f4e4254e5c8b110d1bf47f5e9b747c2f10616ef0fb9f66bca612fc2e3: Status 404 returned error can't find the container with id 7a734c3f4e4254e5c8b110d1bf47f5e9b747c2f10616ef0fb9f66bca612fc2e3 Sep 15 20:54:25.544721 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.544690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" event={"ID":"7463aa39-ce69-4a41-ba13-4fb5f8960df4","Type":"ContainerStarted","Data":"f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d"} Sep 15 20:54:25.545214 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.544744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" event={"ID":"7463aa39-ce69-4a41-ba13-4fb5f8960df4","Type":"ContainerStarted","Data":"7a734c3f4e4254e5c8b110d1bf47f5e9b747c2f10616ef0fb9f66bca612fc2e3"} Sep 15 20:54:25.545214 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.544844 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:54:25.564060 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.564019 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" podStartSLOduration=2.56400127 podStartE2EDuration="2.56400127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:25.562844427 +0000 UTC m=+1062.756797885" watchObservedRunningTime="2026-09-15 20:54:25.56400127 +0000 UTC m=+1062.757954727" Sep 15 20:54:25.738485 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.738452 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl"] Sep 15 20:54:25.745553 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.745531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:25.747844 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.747822 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ded67cd1\"" Sep 15 20:54:25.747973 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.747834 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ded67cd1-dockercfg-bjs24\"" Sep 15 20:54:25.752054 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.752020 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl"] Sep 15 20:54:25.811804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.811652 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl\" (UID: \"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa\") " pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:25.812023 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.811856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmqf2\" (UniqueName: \"kubernetes.io/projected/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-kube-api-access-dmqf2\") pod \"payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl\" (UID: \"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa\") " pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:25.913049 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.913002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqf2\" (UniqueName: \"kubernetes.io/projected/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-kube-api-access-dmqf2\") pod \"payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl\" (UID: \"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa\") " pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:25.913049 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.913065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl\" (UID: \"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa\") " pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:25.913975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.913948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl\" (UID: \"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa\") " pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:25.921100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:25.921076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmqf2\" (UniqueName: \"kubernetes.io/projected/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-kube-api-access-dmqf2\") pod \"payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl\" (UID: \"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa\") " pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:26.061663 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.061632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:26.168006 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.167961 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:54:26.168090 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.168041 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:54:26.168090 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.168071 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:54:26.367330 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.367299 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl"] Sep 15 20:54:26.369302 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:54:26.369265 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc484bc6a_9ba6_4c88_8a49_c7784da8d1aa.slice/crio-04e1b831151fc4e295cbff43ddd29abe62fba9b8b8a47b0a13f285bd59d225a0 WatchSource:0}: Error finding container 04e1b831151fc4e295cbff43ddd29abe62fba9b8b8a47b0a13f285bd59d225a0: Status 404 returned error can't find the container with id 04e1b831151fc4e295cbff43ddd29abe62fba9b8b8a47b0a13f285bd59d225a0 Sep 15 20:54:26.558622 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.558579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" event={"ID":"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877","Type":"ContainerStarted","Data":"825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97"} Sep 15 20:54:26.560511 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.560471 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" event={"ID":"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa","Type":"ContainerStarted","Data":"7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f"} Sep 15 20:54:26.560661 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.560646 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:26.560698 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.560673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" event={"ID":"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa","Type":"ContainerStarted","Data":"04e1b831151fc4e295cbff43ddd29abe62fba9b8b8a47b0a13f285bd59d225a0"} Sep 15 20:54:26.577058 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.577010 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" podStartSLOduration=1.671122682 podStartE2EDuration="4.576993481s" totalImagesPullingTime="2.905870799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:26.575206923 +0000 UTC m=+1063.769160379" watchObservedRunningTime="2026-09-15 20:54:26.576993481 +0000 UTC m=+1063.770946936" Sep 15 20:54:26.589972 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.589921 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" podStartSLOduration=1.589906762 podStartE2EDuration="1.589906762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:26.589148473 +0000 UTC m=+1063.783101928" watchObservedRunningTime="2026-09-15 20:54:26.589906762 +0000 UTC m=+1063.783860216" Sep 15 20:54:26.973364 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:26.973325 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:27.505563 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:27.505528 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 20:54:27.978680 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:27.978650 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:28.571271 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:28.571238 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:28.572100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:28.572079 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:29.513551 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:29.513524 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 20:54:31.068070 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.068038 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh"] Sep 15 20:54:31.071698 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.071683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 20:54:31.073819 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.073796 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-988a09\"" Sep 15 20:54:31.073937 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.073853 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-988a09-dockercfg-2vccj\"" Sep 15 20:54:31.078909 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.078877 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh"] Sep 15 20:54:31.162088 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.162049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584fz\" (UniqueName: \"kubernetes.io/projected/2de27a99-0831-464e-b503-5927b83e8c2d-kube-api-access-584fz\") pod \"payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh\" (UID: \"2de27a99-0831-464e-b503-5927b83e8c2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 20:54:31.162088 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.162090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de27a99-0831-464e-b503-5927b83e8c2d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh\" (UID: \"2de27a99-0831-464e-b503-5927b83e8c2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 20:54:31.263547 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.263491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584fz\" (UniqueName: \"kubernetes.io/projected/2de27a99-0831-464e-b503-5927b83e8c2d-kube-api-access-584fz\") pod \"payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh\" (UID: \"2de27a99-0831-464e-b503-5927b83e8c2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 20:54:31.263723 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.263556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de27a99-0831-464e-b503-5927b83e8c2d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh\" (UID: \"2de27a99-0831-464e-b503-5927b83e8c2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 20:54:31.264194 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.264173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de27a99-0831-464e-b503-5927b83e8c2d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh\" (UID: \"2de27a99-0831-464e-b503-5927b83e8c2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 20:54:31.271032 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.271006 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-584fz\" (UniqueName: \"kubernetes.io/projected/2de27a99-0831-464e-b503-5927b83e8c2d-kube-api-access-584fz\") pod \"payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh\" (UID: \"2de27a99-0831-464e-b503-5927b83e8c2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 20:54:31.384344 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.384267 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 20:54:31.668642 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:31.668617 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh"] Sep 15 20:54:31.670711 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:54:31.670683 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de27a99_0831_464e_b503_5927b83e8c2d.slice/crio-990eea56810d861b923ba035e13eb1cf35f991f94573558b965c4261c3613174 WatchSource:0}: Error finding container 990eea56810d861b923ba035e13eb1cf35f991f94573558b965c4261c3613174: Status 404 returned error can't find the container with id 990eea56810d861b923ba035e13eb1cf35f991f94573558b965c4261c3613174 Sep 15 20:54:32.595300 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:32.595264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" event={"ID":"2de27a99-0831-464e-b503-5927b83e8c2d","Type":"ContainerStarted","Data":"671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950"} Sep 15 20:54:32.595814 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:32.595315 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" event={"ID":"2de27a99-0831-464e-b503-5927b83e8c2d","Type":"ContainerStarted","Data":"990eea56810d861b923ba035e13eb1cf35f991f94573558b965c4261c3613174"} Sep 15 20:54:32.595814 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:32.595345 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 20:54:32.610101 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:32.610045 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" podStartSLOduration=1.6100274620000001 podStartE2EDuration="1.610027462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:32.608948009 +0000 UTC m=+1069.802901464" watchObservedRunningTime="2026-09-15 20:54:32.610027462 +0000 UTC m=+1069.803980917" Sep 15 20:54:36.563473 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:36.563376 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:54:37.141204 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:37.141165 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9"] Sep 15 20:54:37.144896 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:37.144874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9" Sep 15 20:54:37.148109 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:37.148089 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 20:54:37.161967 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:37.161942 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9"] Sep 15 20:54:37.216123 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:37.216087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e1de59-18d8-4fb7-8929-8267d659fadc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9\" (UID: \"99e1de59-18d8-4fb7-8929-8267d659fadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9" Sep 15 20:54:37.317141 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:37.317104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e1de59-18d8-4fb7-8929-8267d659fadc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9\" (UID: \"99e1de59-18d8-4fb7-8929-8267d659fadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9" Sep 15 20:54:37.317783 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:37.317760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e1de59-18d8-4fb7-8929-8267d659fadc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9\" (UID: \"99e1de59-18d8-4fb7-8929-8267d659fadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9" Sep 15 20:54:37.455848 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:37.455775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9" Sep 15 20:54:37.568051 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:37.568028 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:37.751660 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:37.751625 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9"] Sep 15 20:54:38.629256 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:38.629223 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9" event={"ID":"99e1de59-18d8-4fb7-8929-8267d659fadc","Type":"ContainerStarted","Data":"5a2e58d985ed76968a29d88f5c543ec745c17806ee0debf6711e0785b674e7c1"} Sep 15 20:54:42.650591 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:42.650491 2579 generic.go:356] "Generic (PLEG): container finished" podID="99e1de59-18d8-4fb7-8929-8267d659fadc" containerID="d7f7241e017c0acf4c1d5c53853ce4b7ae0523a6b8bb21f36572ec5aa842fbe5" exitCode=0 Sep 15 20:54:42.650957 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:42.650587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9" event={"ID":"99e1de59-18d8-4fb7-8929-8267d659fadc","Type":"ContainerDied","Data":"d7f7241e017c0acf4c1d5c53853ce4b7ae0523a6b8bb21f36572ec5aa842fbe5"} Sep 15 20:54:43.602385 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:43.602351 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 20:54:43.807005 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:43.806977 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9" Sep 15 20:54:43.883539 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:43.883414 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e1de59-18d8-4fb7-8929-8267d659fadc-maas-api-service-ca\") pod \"99e1de59-18d8-4fb7-8929-8267d659fadc\" (UID: \"99e1de59-18d8-4fb7-8929-8267d659fadc\") " Sep 15 20:54:43.883871 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:43.883839 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e1de59-18d8-4fb7-8929-8267d659fadc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99e1de59-18d8-4fb7-8929-8267d659fadc" (UID: "99e1de59-18d8-4fb7-8929-8267d659fadc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:54:43.985141 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:43.985096 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e1de59-18d8-4fb7-8929-8267d659fadc-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:54:44.614670 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:44.614627 2579 patch_prober.go:28] interesting pod/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 15 20:54:44.614838 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:44.614698 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 15 20:54:44.663176 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:44.663143 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9" event={"ID":"99e1de59-18d8-4fb7-8929-8267d659fadc","Type":"ContainerDied","Data":"5a2e58d985ed76968a29d88f5c543ec745c17806ee0debf6711e0785b674e7c1"} Sep 15 20:54:44.663176 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:44.663177 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9" Sep 15 20:54:44.663338 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:44.663200 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a2e58d985ed76968a29d88f5c543ec745c17806ee0debf6711e0785b674e7c1" Sep 15 20:54:45.669252 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:45.669178 2579 generic.go:356] "Generic (PLEG): container finished" podID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerID="825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97" exitCode=0 Sep 15 20:54:45.669252 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:45.669219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" event={"ID":"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877","Type":"ContainerDied","Data":"825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97"} Sep 15 20:54:45.669727 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:45.669600 2579 scope.go:117] "RemoveContainer" containerID="825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97" Sep 15 20:54:45.670777 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:45.670744 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:54:45.670890 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:45.670828 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:54:45.670953 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:45.670903 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:54:46.677262 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:46.677217 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" event={"ID":"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877","Type":"ContainerStarted","Data":"f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625"} Sep 15 20:54:46.973392 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:46.973297 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:46.973582 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:46.973556 2579 patch_prober.go:28] interesting pod/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 15 20:54:46.973646 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:46.973621 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 15 20:54:47.893707 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:47.893672 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl"] Sep 15 20:54:47.894473 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:47.894440 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" podUID="c484bc6a-9ba6-4c88-8a49-c7784da8d1aa" containerName="payload-processing" containerID="cri-o://7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f" gracePeriod=30 Sep 15 20:54:47.973943 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:47.973912 2579 patch_prober.go:28] interesting pod/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 15 20:54:47.974105 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:47.973965 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 15 20:54:48.049883 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.049857 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9"] Sep 15 20:54:48.051823 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.051797 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ded67cd1-6jzr9"] Sep 15 20:54:48.171830 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.171803 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:48.323136 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.323096 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-plugins-config-volume\") pod \"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa\" (UID: \"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa\") " Sep 15 20:54:48.323136 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.323136 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmqf2\" (UniqueName: \"kubernetes.io/projected/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-kube-api-access-dmqf2\") pod \"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa\" (UID: \"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa\") " Sep 15 20:54:48.323568 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.323544 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c484bc6a-9ba6-4c88-8a49-c7784da8d1aa" (UID: "c484bc6a-9ba6-4c88-8a49-c7784da8d1aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:54:48.325559 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.325531 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-kube-api-access-dmqf2" (OuterVolumeSpecName: "kube-api-access-dmqf2") pod "c484bc6a-9ba6-4c88-8a49-c7784da8d1aa" (UID: "c484bc6a-9ba6-4c88-8a49-c7784da8d1aa"). InnerVolumeSpecName "kube-api-access-dmqf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:54:48.424862 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.424767 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:54:48.424862 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.424800 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmqf2\" (UniqueName: \"kubernetes.io/projected/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa-kube-api-access-dmqf2\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:54:48.695803 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.695700 2579 generic.go:356] "Generic (PLEG): container finished" podID="c484bc6a-9ba6-4c88-8a49-c7784da8d1aa" containerID="7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f" exitCode=0 Sep 15 20:54:48.695803 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.695756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" event={"ID":"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa","Type":"ContainerDied","Data":"7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f"} Sep 15 20:54:48.696199 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.695807 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" event={"ID":"c484bc6a-9ba6-4c88-8a49-c7784da8d1aa","Type":"ContainerDied","Data":"04e1b831151fc4e295cbff43ddd29abe62fba9b8b8a47b0a13f285bd59d225a0"} Sep 15 20:54:48.696199 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.695838 2579 scope.go:117] "RemoveContainer" containerID="7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f" Sep 15 20:54:48.696199 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.696006 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl" Sep 15 20:54:48.710437 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.710414 2579 scope.go:117] "RemoveContainer" containerID="7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f" Sep 15 20:54:48.710739 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:54:48.710718 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f\": container with ID starting with 7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f not found: ID does not exist" containerID="7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f" Sep 15 20:54:48.710816 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.710748 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f"} err="failed to get container status \"7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f\": rpc error: code = NotFound desc = could not find container \"7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f\": container with ID starting with 7e0c901d4e2bf0f5bf831cb6ce2d06f769c4c7bbd5c508ab5db07dd1ca367c0f not found: ID does not exist" Sep 15 20:54:48.720911 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.720885 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl"] Sep 15 20:54:48.722565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.722542 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ded67cd1-666d8d97f5-74cxl"] Sep 15 20:54:48.973122 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.973027 2579 patch_prober.go:28] interesting pod/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 15 20:54:48.973122 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:48.973075 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 15 20:54:49.230388 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.230293 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf"] Sep 15 20:54:49.230714 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.230699 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c484bc6a-9ba6-4c88-8a49-c7784da8d1aa" containerName="payload-processing" Sep 15 20:54:49.230714 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.230714 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c484bc6a-9ba6-4c88-8a49-c7784da8d1aa" containerName="payload-processing" Sep 15 20:54:49.230837 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.230731 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e1de59-18d8-4fb7-8929-8267d659fadc" containerName="revoke-keys" Sep 15 20:54:49.230837 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.230737 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e1de59-18d8-4fb7-8929-8267d659fadc" containerName="revoke-keys" Sep 15 20:54:49.230837 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.230789 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c484bc6a-9ba6-4c88-8a49-c7784da8d1aa" containerName="payload-processing" Sep 15 20:54:49.230837 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.230799 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e1de59-18d8-4fb7-8929-8267d659fadc" containerName="revoke-keys" Sep 15 20:54:49.235762 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.235742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.238194 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.238176 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e8af3fa9-gw-openshift-default-dockercfg-6mv8z\"" Sep 15 20:54:49.244867 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.244843 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf"] Sep 15 20:54:49.312884 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.312854 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e1de59-18d8-4fb7-8929-8267d659fadc" path="/var/lib/kubelet/pods/99e1de59-18d8-4fb7-8929-8267d659fadc/volumes" Sep 15 20:54:49.313223 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.313210 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c484bc6a-9ba6-4c88-8a49-c7784da8d1aa" path="/var/lib/kubelet/pods/c484bc6a-9ba6-4c88-8a49-c7784da8d1aa/volumes" Sep 15 20:54:49.334153 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.334129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-certs\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.334313 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.334158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-data\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.334313 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.334188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l998\" (UniqueName: \"kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-kube-api-access-4l998\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.334313 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.334290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-credential-socket\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.334492 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.334340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-podinfo\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.334492 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.334369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-socket\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.334492 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.334399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0209f6d-4426-4d66-8657-7885ccfbf026-istiod-ca-cert\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.334492 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.334428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-envoy\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.334492 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.334475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-token\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.435617 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.435575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-envoy\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.435617 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.435623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-token\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.435895 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.435674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-certs\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.435895 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.435688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-data\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.435895 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.435719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l998\" (UniqueName: \"kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-kube-api-access-4l998\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.435895 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.435750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-credential-socket\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.436154 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.436088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-podinfo\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.436154 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.436131 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-certs\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.436267 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.436136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-socket\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.436267 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.436183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-credential-socket\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.436267 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.436189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0209f6d-4426-4d66-8657-7885ccfbf026-istiod-ca-cert\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.436267 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.436238 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-data\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.436582 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.436490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-socket\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.436731 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.436715 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0209f6d-4426-4d66-8657-7885ccfbf026-istiod-ca-cert\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.438136 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.438107 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-envoy\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.438649 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.438631 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-podinfo\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.443956 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.443938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-token\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.444029 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.443969 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l998\" (UniqueName: \"kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-kube-api-access-4l998\") pod \"e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.547336 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.547303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:49.841760 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.841688 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf"] Sep 15 20:54:49.845103 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:54:49.845077 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0209f6d_4426_4d66_8657_7885ccfbf026.slice/crio-93bee18f25254d6bc0fd327b6e8c3000af05c5cb2c9dbad5e295095197fe8421 WatchSource:0}: Error finding container 93bee18f25254d6bc0fd327b6e8c3000af05c5cb2c9dbad5e295095197fe8421: Status 404 returned error can't find the container with id 93bee18f25254d6bc0fd327b6e8c3000af05c5cb2c9dbad5e295095197fe8421 Sep 15 20:54:49.847823 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.847794 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:54:49.847903 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.847860 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:54:49.847903 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.847889 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:54:49.973972 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.973937 2579 patch_prober.go:28] interesting pod/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 15 20:54:49.974426 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:49.973996 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 15 20:54:50.712245 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:50.712199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" event={"ID":"c0209f6d-4426-4d66-8657-7885ccfbf026","Type":"ContainerStarted","Data":"10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5"} Sep 15 20:54:50.712434 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:50.712253 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" event={"ID":"c0209f6d-4426-4d66-8657-7885ccfbf026","Type":"ContainerStarted","Data":"93bee18f25254d6bc0fd327b6e8c3000af05c5cb2c9dbad5e295095197fe8421"} Sep 15 20:54:50.729823 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:50.729768 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" podStartSLOduration=1.729749202 podStartE2EDuration="1.729749202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:50.729233694 +0000 UTC m=+1087.923187150" watchObservedRunningTime="2026-09-15 20:54:50.729749202 +0000 UTC m=+1087.923702659" Sep 15 20:54:50.974021 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:50.973933 2579 patch_prober.go:28] interesting pod/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 15 20:54:50.974021 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:50.973996 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 15 20:54:51.548150 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:51.548114 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:51.553684 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:51.553656 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:51.718113 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:51.718071 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:51.719081 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:51.719054 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:54:51.977823 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:51.977742 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:51.978195 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:51.978136 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:51.979053 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:51.979033 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 20:54:53.389751 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.389719 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx"] Sep 15 20:54:53.398315 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.398298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:54:53.400713 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.400670 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e8af3fa9-dockercfg-ksvxx\"" Sep 15 20:54:53.400870 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.400845 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e8af3fa9\"" Sep 15 20:54:53.402622 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.402601 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx"] Sep 15 20:54:53.422040 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.422011 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg"] Sep 15 20:54:53.425491 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.425471 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:54:53.432995 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.432968 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg"] Sep 15 20:54:53.475699 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.475661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2dfd9b-ae63-4895-9279-493e0db90be8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg\" (UID: \"1d2dfd9b-ae63-4895-9279-493e0db90be8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:54:53.475849 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.475716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nplbp\" (UniqueName: \"kubernetes.io/projected/1d2dfd9b-ae63-4895-9279-493e0db90be8-kube-api-access-nplbp\") pod \"payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg\" (UID: \"1d2dfd9b-ae63-4895-9279-493e0db90be8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:54:53.475901 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.475860 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4vb\" (UniqueName: \"kubernetes.io/projected/9720d693-7cf9-43ad-9c63-40f2cff2359f-kube-api-access-kb4vb\") pod \"payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx\" (UID: \"9720d693-7cf9-43ad-9c63-40f2cff2359f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:54:53.475994 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.475975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9720d693-7cf9-43ad-9c63-40f2cff2359f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx\" (UID: \"9720d693-7cf9-43ad-9c63-40f2cff2359f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:54:53.577005 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.576970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9720d693-7cf9-43ad-9c63-40f2cff2359f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx\" (UID: \"9720d693-7cf9-43ad-9c63-40f2cff2359f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:54:53.577189 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.577015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2dfd9b-ae63-4895-9279-493e0db90be8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg\" (UID: \"1d2dfd9b-ae63-4895-9279-493e0db90be8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:54:53.577189 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.577037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nplbp\" (UniqueName: \"kubernetes.io/projected/1d2dfd9b-ae63-4895-9279-493e0db90be8-kube-api-access-nplbp\") pod \"payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg\" (UID: \"1d2dfd9b-ae63-4895-9279-493e0db90be8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:54:53.577189 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.577100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4vb\" (UniqueName: \"kubernetes.io/projected/9720d693-7cf9-43ad-9c63-40f2cff2359f-kube-api-access-kb4vb\") pod \"payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx\" (UID: \"9720d693-7cf9-43ad-9c63-40f2cff2359f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:54:53.577711 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.577691 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9720d693-7cf9-43ad-9c63-40f2cff2359f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx\" (UID: \"9720d693-7cf9-43ad-9c63-40f2cff2359f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:54:53.577773 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.577752 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2dfd9b-ae63-4895-9279-493e0db90be8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg\" (UID: \"1d2dfd9b-ae63-4895-9279-493e0db90be8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:54:53.584736 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.584713 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nplbp\" (UniqueName: \"kubernetes.io/projected/1d2dfd9b-ae63-4895-9279-493e0db90be8-kube-api-access-nplbp\") pod \"payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg\" (UID: \"1d2dfd9b-ae63-4895-9279-493e0db90be8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:54:53.584855 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.584766 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4vb\" (UniqueName: \"kubernetes.io/projected/9720d693-7cf9-43ad-9c63-40f2cff2359f-kube-api-access-kb4vb\") pod \"payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx\" (UID: \"9720d693-7cf9-43ad-9c63-40f2cff2359f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:54:53.710390 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.710304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:54:53.736705 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.736674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:54:53.990510 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:53.990473 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx"] Sep 15 20:54:53.991980 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:54:53.991954 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9720d693_7cf9_43ad_9c63_40f2cff2359f.slice/crio-fdca7661757aa549e92b16fcb70883fede96f29066288012a876f864ec3f4f8f WatchSource:0}: Error finding container fdca7661757aa549e92b16fcb70883fede96f29066288012a876f864ec3f4f8f: Status 404 returned error can't find the container with id fdca7661757aa549e92b16fcb70883fede96f29066288012a876f864ec3f4f8f Sep 15 20:54:54.020012 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:54.019987 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg"] Sep 15 20:54:54.022990 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:54:54.022957 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2dfd9b_ae63_4895_9279_493e0db90be8.slice/crio-7929f3d7440e03259e159c63062c9f08b1fec70a98a5ae1d8e8ce468cb15ee90 WatchSource:0}: Error finding container 7929f3d7440e03259e159c63062c9f08b1fec70a98a5ae1d8e8ce468cb15ee90: Status 404 returned error can't find the container with id 7929f3d7440e03259e159c63062c9f08b1fec70a98a5ae1d8e8ce468cb15ee90 Sep 15 20:54:54.735128 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:54.735079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" event={"ID":"1d2dfd9b-ae63-4895-9279-493e0db90be8","Type":"ContainerStarted","Data":"7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f"} Sep 15 20:54:54.735623 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:54.735151 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:54:54.735623 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:54.735176 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" event={"ID":"1d2dfd9b-ae63-4895-9279-493e0db90be8","Type":"ContainerStarted","Data":"7929f3d7440e03259e159c63062c9f08b1fec70a98a5ae1d8e8ce468cb15ee90"} Sep 15 20:54:54.736899 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:54.736870 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" event={"ID":"9720d693-7cf9-43ad-9c63-40f2cff2359f","Type":"ContainerStarted","Data":"44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85"} Sep 15 20:54:54.737038 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:54.736914 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" event={"ID":"9720d693-7cf9-43ad-9c63-40f2cff2359f","Type":"ContainerStarted","Data":"fdca7661757aa549e92b16fcb70883fede96f29066288012a876f864ec3f4f8f"} Sep 15 20:54:54.737038 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:54.736933 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:54:54.749842 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:54.749793 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" podStartSLOduration=1.749776023 podStartE2EDuration="1.749776023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:54.749228002 +0000 UTC m=+1091.943181460" watchObservedRunningTime="2026-09-15 20:54:54.749776023 +0000 UTC m=+1091.943729480" Sep 15 20:54:54.763182 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:54:54.763134 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" podStartSLOduration=1.7631198449999999 podStartE2EDuration="1.763119845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:54.762333205 +0000 UTC m=+1091.956286660" watchObservedRunningTime="2026-09-15 20:54:54.763119845 +0000 UTC m=+1091.957073301" Sep 15 20:55:04.956635 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:04.956604 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw"] Sep 15 20:55:04.960743 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:04.960721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:04.964003 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:04.963972 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"simulator-w4-16bde4-kserve-self-signed-certs\"" Sep 15 20:55:04.964131 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:04.963984 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"default-dockercfg-sljmb\"" Sep 15 20:55:04.964131 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:04.964039 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"openshift-service-ca.crt\"" Sep 15 20:55:04.964131 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:04.964054 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"kube-root-ca.crt\"" Sep 15 20:55:04.970615 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:04.970588 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw"] Sep 15 20:55:05.093935 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.093892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-dshm\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.094247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.094227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jltf2\" (UniqueName: \"kubernetes.io/projected/9afe40ed-514d-4fd3-a64b-4c48549890a3-kube-api-access-jltf2\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.094451 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.094434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-tmp-dir\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.094688 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.094654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afe40ed-514d-4fd3-a64b-4c48549890a3-tls-certs\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.094868 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.094842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-model-cache\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.094977 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.094899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-home\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.196575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.195550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-dshm\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.196575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.195604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jltf2\" (UniqueName: \"kubernetes.io/projected/9afe40ed-514d-4fd3-a64b-4c48549890a3-kube-api-access-jltf2\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.196575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.195653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-tmp-dir\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.196575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.195707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afe40ed-514d-4fd3-a64b-4c48549890a3-tls-certs\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.196575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.195736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-model-cache\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.196575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.195777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-home\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.196575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.196230 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-home\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.197542 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.197523 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-tmp-dir\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.197855 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.197808 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-model-cache\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.200687 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.200663 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afe40ed-514d-4fd3-a64b-4c48549890a3-tls-certs\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.200687 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.200673 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-dshm\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.204823 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.204795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jltf2\" (UniqueName: \"kubernetes.io/projected/9afe40ed-514d-4fd3-a64b-4c48549890a3-kube-api-access-jltf2\") pod \"simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.271903 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.271878 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:05.561803 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.561712 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw"] Sep 15 20:55:05.564719 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:55:05.564688 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9afe40ed_514d_4fd3_a64b_4c48549890a3.slice/crio-cfbbadd23462c16e23695d2a9aa4ab2cd9bc98729d281a7f5b1d35cb9b5eef8b WatchSource:0}: Error finding container cfbbadd23462c16e23695d2a9aa4ab2cd9bc98729d281a7f5b1d35cb9b5eef8b: Status 404 returned error can't find the container with id cfbbadd23462c16e23695d2a9aa4ab2cd9bc98729d281a7f5b1d35cb9b5eef8b Sep 15 20:55:05.744131 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.744103 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:55:05.744298 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.744168 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:55:05.811720 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.811678 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" event={"ID":"9afe40ed-514d-4fd3-a64b-4c48549890a3","Type":"ContainerStarted","Data":"b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84"} Sep 15 20:55:05.811942 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.811727 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" event={"ID":"9afe40ed-514d-4fd3-a64b-4c48549890a3","Type":"ContainerStarted","Data":"cfbbadd23462c16e23695d2a9aa4ab2cd9bc98729d281a7f5b1d35cb9b5eef8b"} Sep 15 20:55:05.827338 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:05.827289 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" podStartSLOduration=1.8272711780000002 podStartE2EDuration="1.827271178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:55:05.82682163 +0000 UTC m=+1103.020775086" watchObservedRunningTime="2026-09-15 20:55:05.827271178 +0000 UTC m=+1103.021224632" Sep 15 20:55:13.859203 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:13.859171 2579 generic.go:356] "Generic (PLEG): container finished" podID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerID="10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5" exitCode=0 Sep 15 20:55:13.859622 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:13.859213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" event={"ID":"c0209f6d-4426-4d66-8657-7885ccfbf026","Type":"ContainerDied","Data":"10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5"} Sep 15 20:55:13.859622 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:13.859589 2579 scope.go:117] "RemoveContainer" containerID="10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5" Sep 15 20:55:13.860639 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:13.860612 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:55:13.860718 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:13.860676 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:55:13.860718 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:13.860706 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:55:14.865632 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:14.865600 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" event={"ID":"c0209f6d-4426-4d66-8657-7885ccfbf026","Type":"ContainerStarted","Data":"a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8"} Sep 15 20:55:15.272593 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:15.272564 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:15.272757 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:15.272657 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:15.311867 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:15.311837 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:15.547881 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:15.547800 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:55:15.548032 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:15.547982 2579 patch_prober.go:28] interesting pod/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 15 20:55:15.548075 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:15.548027 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 15 20:55:15.906538 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:15.906446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 20:55:16.548344 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:16.548306 2579 patch_prober.go:28] interesting pod/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 15 20:55:16.548554 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:16.548363 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 15 20:55:17.252492 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.252458 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c"] Sep 15 20:55:17.256342 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.256326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c" Sep 15 20:55:17.258184 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.258162 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 20:55:17.263151 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.263128 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c"] Sep 15 20:55:17.314173 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.314146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad768b-6446-4540-9ca9-09a24bd89621-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c\" (UID: \"b9ad768b-6446-4540-9ca9-09a24bd89621\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c" Sep 15 20:55:17.415240 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.415196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad768b-6446-4540-9ca9-09a24bd89621-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c\" (UID: \"b9ad768b-6446-4540-9ca9-09a24bd89621\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c" Sep 15 20:55:17.415852 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.415831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad768b-6446-4540-9ca9-09a24bd89621-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c\" (UID: \"b9ad768b-6446-4540-9ca9-09a24bd89621\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c" Sep 15 20:55:17.548209 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.548174 2579 patch_prober.go:28] interesting pod/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 15 20:55:17.548409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.548236 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 15 20:55:17.567769 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.567741 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c" Sep 15 20:55:17.857323 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.857283 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c"] Sep 15 20:55:17.859406 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:55:17.859380 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ad768b_6446_4540_9ca9_09a24bd89621.slice/crio-8a0c580beb77c2e0e60dec7d19fdcd658a4cbd86d01d367e4a46bc027d8bcb27 WatchSource:0}: Error finding container 8a0c580beb77c2e0e60dec7d19fdcd658a4cbd86d01d367e4a46bc027d8bcb27: Status 404 returned error can't find the container with id 8a0c580beb77c2e0e60dec7d19fdcd658a4cbd86d01d367e4a46bc027d8bcb27 Sep 15 20:55:17.900526 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:17.897897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c" event={"ID":"b9ad768b-6446-4540-9ca9-09a24bd89621","Type":"ContainerStarted","Data":"8a0c580beb77c2e0e60dec7d19fdcd658a4cbd86d01d367e4a46bc027d8bcb27"} Sep 15 20:55:18.548300 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:18.548223 2579 patch_prober.go:28] interesting pod/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 15 20:55:18.548300 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:18.548277 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 15 20:55:18.904694 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:18.904655 2579 generic.go:356] "Generic (PLEG): container finished" podID="b9ad768b-6446-4540-9ca9-09a24bd89621" containerID="10445952fd22dfcbc9a8522f2a76224d80996a4fc64432f3bf4c664b21ff4e9d" exitCode=0 Sep 15 20:55:18.904866 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:18.904706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c" event={"ID":"b9ad768b-6446-4540-9ca9-09a24bd89621","Type":"ContainerDied","Data":"10445952fd22dfcbc9a8522f2a76224d80996a4fc64432f3bf4c664b21ff4e9d"} Sep 15 20:55:19.548217 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:19.548181 2579 patch_prober.go:28] interesting pod/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 15 20:55:19.548409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:19.548241 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 15 20:55:20.041826 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:20.041800 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c" Sep 15 20:55:20.140660 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:20.140630 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad768b-6446-4540-9ca9-09a24bd89621-maas-api-service-ca\") pod \"b9ad768b-6446-4540-9ca9-09a24bd89621\" (UID: \"b9ad768b-6446-4540-9ca9-09a24bd89621\") " Sep 15 20:55:20.140994 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:20.140970 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ad768b-6446-4540-9ca9-09a24bd89621-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9ad768b-6446-4540-9ca9-09a24bd89621" (UID: "b9ad768b-6446-4540-9ca9-09a24bd89621"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:55:20.241341 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:20.241257 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad768b-6446-4540-9ca9-09a24bd89621-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:20.552112 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:20.552081 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:55:20.552492 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:20.552369 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:55:20.553108 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:20.553087 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:55:20.916425 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:20.916343 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c" Sep 15 20:55:20.916425 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:20.916347 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c" event={"ID":"b9ad768b-6446-4540-9ca9-09a24bd89621","Type":"ContainerDied","Data":"8a0c580beb77c2e0e60dec7d19fdcd658a4cbd86d01d367e4a46bc027d8bcb27"} Sep 15 20:55:20.916425 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:20.916413 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a0c580beb77c2e0e60dec7d19fdcd658a4cbd86d01d367e4a46bc027d8bcb27" Sep 15 20:55:24.054638 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.054606 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg"] Sep 15 20:55:24.055094 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.054853 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" podUID="1d2dfd9b-ae63-4895-9279-493e0db90be8" containerName="payload-processing" containerID="cri-o://7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f" gracePeriod=30 Sep 15 20:55:24.062385 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.062335 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx"] Sep 15 20:55:24.062633 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.062562 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" podUID="9720d693-7cf9-43ad-9c63-40f2cff2359f" containerName="payload-pre-processing" containerID="cri-o://44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85" gracePeriod=30 Sep 15 20:55:24.213789 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.213752 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c"] Sep 15 20:55:24.218281 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.218250 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8af3fa9-t5t2c"] Sep 15 20:55:24.317135 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.317066 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:55:24.348533 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.346332 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:55:24.349372 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.349343 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf"] Sep 15 20:55:24.349658 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.349601 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" containerID="cri-o://a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8" gracePeriod=30 Sep 15 20:55:24.377308 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.377276 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2dfd9b-ae63-4895-9279-493e0db90be8-plugins-config-volume\") pod \"1d2dfd9b-ae63-4895-9279-493e0db90be8\" (UID: \"1d2dfd9b-ae63-4895-9279-493e0db90be8\") " Sep 15 20:55:24.377509 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.377405 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nplbp\" (UniqueName: \"kubernetes.io/projected/1d2dfd9b-ae63-4895-9279-493e0db90be8-kube-api-access-nplbp\") pod \"1d2dfd9b-ae63-4895-9279-493e0db90be8\" (UID: \"1d2dfd9b-ae63-4895-9279-493e0db90be8\") " Sep 15 20:55:24.377509 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.377427 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9720d693-7cf9-43ad-9c63-40f2cff2359f-plugins-config-volume\") pod \"9720d693-7cf9-43ad-9c63-40f2cff2359f\" (UID: \"9720d693-7cf9-43ad-9c63-40f2cff2359f\") " Sep 15 20:55:24.377509 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.377456 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb4vb\" (UniqueName: \"kubernetes.io/projected/9720d693-7cf9-43ad-9c63-40f2cff2359f-kube-api-access-kb4vb\") pod \"9720d693-7cf9-43ad-9c63-40f2cff2359f\" (UID: \"9720d693-7cf9-43ad-9c63-40f2cff2359f\") " Sep 15 20:55:24.377777 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.377752 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2dfd9b-ae63-4895-9279-493e0db90be8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d2dfd9b-ae63-4895-9279-493e0db90be8" (UID: "1d2dfd9b-ae63-4895-9279-493e0db90be8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:55:24.378135 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.378071 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2dfd9b-ae63-4895-9279-493e0db90be8-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:24.378135 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.378072 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9720d693-7cf9-43ad-9c63-40f2cff2359f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9720d693-7cf9-43ad-9c63-40f2cff2359f" (UID: "9720d693-7cf9-43ad-9c63-40f2cff2359f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:55:24.379676 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.379645 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2dfd9b-ae63-4895-9279-493e0db90be8-kube-api-access-nplbp" (OuterVolumeSpecName: "kube-api-access-nplbp") pod "1d2dfd9b-ae63-4895-9279-493e0db90be8" (UID: "1d2dfd9b-ae63-4895-9279-493e0db90be8"). InnerVolumeSpecName "kube-api-access-nplbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:55:24.379872 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.379850 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9720d693-7cf9-43ad-9c63-40f2cff2359f-kube-api-access-kb4vb" (OuterVolumeSpecName: "kube-api-access-kb4vb") pod "9720d693-7cf9-43ad-9c63-40f2cff2359f" (UID: "9720d693-7cf9-43ad-9c63-40f2cff2359f"). InnerVolumeSpecName "kube-api-access-kb4vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:55:24.479189 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.479155 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nplbp\" (UniqueName: \"kubernetes.io/projected/1d2dfd9b-ae63-4895-9279-493e0db90be8-kube-api-access-nplbp\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:24.479189 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.479184 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9720d693-7cf9-43ad-9c63-40f2cff2359f-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:24.479189 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.479193 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4vb\" (UniqueName: \"kubernetes.io/projected/9720d693-7cf9-43ad-9c63-40f2cff2359f-kube-api-access-kb4vb\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:24.938094 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.938055 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d2dfd9b-ae63-4895-9279-493e0db90be8" containerID="7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f" exitCode=0 Sep 15 20:55:24.938289 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.938104 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" Sep 15 20:55:24.938289 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.938135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" event={"ID":"1d2dfd9b-ae63-4895-9279-493e0db90be8","Type":"ContainerDied","Data":"7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f"} Sep 15 20:55:24.938289 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.938189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg" event={"ID":"1d2dfd9b-ae63-4895-9279-493e0db90be8","Type":"ContainerDied","Data":"7929f3d7440e03259e159c63062c9f08b1fec70a98a5ae1d8e8ce468cb15ee90"} Sep 15 20:55:24.938289 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.938218 2579 scope.go:117] "RemoveContainer" containerID="7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f" Sep 15 20:55:24.939627 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.939609 2579 generic.go:356] "Generic (PLEG): container finished" podID="9720d693-7cf9-43ad-9c63-40f2cff2359f" containerID="44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85" exitCode=0 Sep 15 20:55:24.939709 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.939659 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" Sep 15 20:55:24.939709 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.939694 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" event={"ID":"9720d693-7cf9-43ad-9c63-40f2cff2359f","Type":"ContainerDied","Data":"44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85"} Sep 15 20:55:24.939802 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.939736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx" event={"ID":"9720d693-7cf9-43ad-9c63-40f2cff2359f","Type":"ContainerDied","Data":"fdca7661757aa549e92b16fcb70883fede96f29066288012a876f864ec3f4f8f"} Sep 15 20:55:24.948278 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.948213 2579 scope.go:117] "RemoveContainer" containerID="7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f" Sep 15 20:55:24.948540 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:55:24.948518 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f\": container with ID starting with 7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f not found: ID does not exist" containerID="7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f" Sep 15 20:55:24.948645 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.948544 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f"} err="failed to get container status \"7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f\": rpc error: code = NotFound desc = could not find container \"7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f\": container with ID starting with 7f91729ec1b1c131fdf6fa1c3ae23232e550f98f59d993164fbe337eb14b885f not found: ID does not exist" Sep 15 20:55:24.948645 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.948562 2579 scope.go:117] "RemoveContainer" containerID="44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85" Sep 15 20:55:24.956542 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.956520 2579 scope.go:117] "RemoveContainer" containerID="44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85" Sep 15 20:55:24.956804 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:55:24.956785 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85\": container with ID starting with 44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85 not found: ID does not exist" containerID="44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85" Sep 15 20:55:24.956856 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.956812 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85"} err="failed to get container status \"44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85\": rpc error: code = NotFound desc = could not find container \"44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85\": container with ID starting with 44d84224b66b944870b3c40c4f53613b30ac687001d46d15b1648aff2058ee85 not found: ID does not exist" Sep 15 20:55:24.960535 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.960517 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg"] Sep 15 20:55:24.965902 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.965884 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e8af3fa9-6cb4d7886-hp2pg"] Sep 15 20:55:24.974740 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.974721 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx"] Sep 15 20:55:24.978225 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:24.978208 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e8af3fa9-655897cdfd-7k2xx"] Sep 15 20:55:25.132441 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.132411 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9"] Sep 15 20:55:25.132912 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.132810 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9720d693-7cf9-43ad-9c63-40f2cff2359f" containerName="payload-pre-processing" Sep 15 20:55:25.132912 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.132823 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9720d693-7cf9-43ad-9c63-40f2cff2359f" containerName="payload-pre-processing" Sep 15 20:55:25.132912 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.132845 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d2dfd9b-ae63-4895-9279-493e0db90be8" containerName="payload-processing" Sep 15 20:55:25.132912 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.132851 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d2dfd9b-ae63-4895-9279-493e0db90be8" containerName="payload-processing" Sep 15 20:55:25.132912 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.132860 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ad768b-6446-4540-9ca9-09a24bd89621" containerName="revoke-keys" Sep 15 20:55:25.132912 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.132865 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ad768b-6446-4540-9ca9-09a24bd89621" containerName="revoke-keys" Sep 15 20:55:25.133286 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.132923 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9720d693-7cf9-43ad-9c63-40f2cff2359f" containerName="payload-pre-processing" Sep 15 20:55:25.133286 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.132930 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d2dfd9b-ae63-4895-9279-493e0db90be8" containerName="payload-processing" Sep 15 20:55:25.133286 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.132940 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ad768b-6446-4540-9ca9-09a24bd89621" containerName="revoke-keys" Sep 15 20:55:25.137318 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.137301 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.142176 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.142160 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0143258d-openshift-default-dockercfg-mck9m\"" Sep 15 20:55:25.155044 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.155023 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9"] Sep 15 20:55:25.188075 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.188048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-envoy\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.188199 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.188083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djw29\" (UniqueName: \"kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-kube-api-access-djw29\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.188199 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.188102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-data\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.188289 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.188211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-credential-socket\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.188289 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.188246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-socket\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.188289 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.188268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-certs\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.188469 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.188362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-podinfo\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.188595 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.188478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-token\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.188595 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.188563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad0394b-adea-4d97-a372-afb5d51ffaec-istiod-ca-cert\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289042 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-token\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad0394b-adea-4d97-a372-afb5d51ffaec-istiod-ca-cert\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-envoy\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djw29\" (UniqueName: \"kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-kube-api-access-djw29\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-data\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289207 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-credential-socket\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289535 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-socket\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289535 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-certs\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289649 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-podinfo\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289649 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-credential-socket\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289740 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289648 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-data\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289792 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-socket\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289868 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-certs\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.289932 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.289873 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad0394b-adea-4d97-a372-afb5d51ffaec-istiod-ca-cert\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.291672 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.291653 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-envoy\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.292075 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.292055 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-podinfo\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.300997 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.300979 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-token\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.301524 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.301484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djw29\" (UniqueName: \"kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-kube-api-access-djw29\") pod \"e2e-ait-0143258d-openshift-default-df7988478-d2tg9\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.312177 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.312153 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d2dfd9b-ae63-4895-9279-493e0db90be8" path="/var/lib/kubelet/pods/1d2dfd9b-ae63-4895-9279-493e0db90be8/volumes" Sep 15 20:55:25.312555 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.312540 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9720d693-7cf9-43ad-9c63-40f2cff2359f" path="/var/lib/kubelet/pods/9720d693-7cf9-43ad-9c63-40f2cff2359f/volumes" Sep 15 20:55:25.312831 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.312818 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ad768b-6446-4540-9ca9-09a24bd89621" path="/var/lib/kubelet/pods/b9ad768b-6446-4540-9ca9-09a24bd89621/volumes" Sep 15 20:55:25.446596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.446480 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:25.723217 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.723193 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9"] Sep 15 20:55:25.725021 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:55:25.724994 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad0394b_adea_4d97_a372_afb5d51ffaec.slice/crio-3d244b454027337c5846037974acf519df6d299d48220c7c6681dd2b44b75fd8 WatchSource:0}: Error finding container 3d244b454027337c5846037974acf519df6d299d48220c7c6681dd2b44b75fd8: Status 404 returned error can't find the container with id 3d244b454027337c5846037974acf519df6d299d48220c7c6681dd2b44b75fd8 Sep 15 20:55:25.727404 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.727372 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:55:25.727469 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.727452 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:55:25.727543 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.727526 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:55:25.956683 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.956651 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" event={"ID":"9ad0394b-adea-4d97-a372-afb5d51ffaec","Type":"ContainerStarted","Data":"9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b"} Sep 15 20:55:25.956905 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.956720 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" event={"ID":"9ad0394b-adea-4d97-a372-afb5d51ffaec","Type":"ContainerStarted","Data":"3d244b454027337c5846037974acf519df6d299d48220c7c6681dd2b44b75fd8"} Sep 15 20:55:25.977832 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:25.977713 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" podStartSLOduration=0.977693751 podStartE2EDuration="977.693751ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:55:25.975959224 +0000 UTC m=+1123.169912693" watchObservedRunningTime="2026-09-15 20:55:25.977693751 +0000 UTC m=+1123.171647207" Sep 15 20:55:26.156014 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.155983 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq"] Sep 15 20:55:26.159578 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.159562 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.161979 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.161959 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b9ff68\"/\"kube-root-ca.crt\"" Sep 15 20:55:26.162106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.162062 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b9ff68\"/\"default-dockercfg-qshwv\"" Sep 15 20:55:26.162106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.162062 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b9ff68\"/\"autores-model-b9ff68-kserve-self-signed-certs\"" Sep 15 20:55:26.162196 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.162104 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b9ff68\"/\"openshift-service-ca.crt\"" Sep 15 20:55:26.169571 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.169545 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq"] Sep 15 20:55:26.197561 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.197535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-home\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.197561 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.197563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lg8v\" (UniqueName: \"kubernetes.io/projected/4a04c99b-a1b2-4c70-9186-034a6c79f81b-kube-api-access-8lg8v\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.197716 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.197583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-dshm\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.197716 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.197604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tmp-dir\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.197716 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.197689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tls-certs\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.197824 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.197719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-model-cache\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.298898 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.298870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-home\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.299069 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.298902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lg8v\" (UniqueName: \"kubernetes.io/projected/4a04c99b-a1b2-4c70-9186-034a6c79f81b-kube-api-access-8lg8v\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.299069 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.298929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-dshm\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.299069 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.298966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tmp-dir\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.299069 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.299011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tls-certs\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.299069 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.299040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-model-cache\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.299297 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.299286 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-home\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.299367 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.299345 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tmp-dir\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.299465 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.299447 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-model-cache\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.301378 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.301359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-dshm\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.301649 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.301631 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tls-certs\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.310421 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.310395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lg8v\" (UniqueName: \"kubernetes.io/projected/4a04c99b-a1b2-4c70-9186-034a6c79f81b-kube-api-access-8lg8v\") pod \"autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.447062 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.447027 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:26.452530 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.452489 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:26.469473 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.469449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:26.739053 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.739027 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq"] Sep 15 20:55:26.740951 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:55:26.740925 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a04c99b_a1b2_4c70_9186_034a6c79f81b.slice/crio-0c2d1ef79af28bbd5a67b1dbe372f4b31927194f74c3c275a361a1ef7ecf71b4 WatchSource:0}: Error finding container 0c2d1ef79af28bbd5a67b1dbe372f4b31927194f74c3c275a361a1ef7ecf71b4: Status 404 returned error can't find the container with id 0c2d1ef79af28bbd5a67b1dbe372f4b31927194f74c3c275a361a1ef7ecf71b4 Sep 15 20:55:26.963111 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.963073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" event={"ID":"4a04c99b-a1b2-4c70-9186-034a6c79f81b","Type":"ContainerStarted","Data":"fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30"} Sep 15 20:55:26.963111 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.963117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" event={"ID":"4a04c99b-a1b2-4c70-9186-034a6c79f81b","Type":"ContainerStarted","Data":"0c2d1ef79af28bbd5a67b1dbe372f4b31927194f74c3c275a361a1ef7ecf71b4"} Sep 15 20:55:26.963565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.963548 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:26.964491 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.964474 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:26.982236 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:26.982189 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" podStartSLOduration=0.982175639 podStartE2EDuration="982.175639ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:55:26.980141779 +0000 UTC m=+1124.174095234" watchObservedRunningTime="2026-09-15 20:55:26.982175639 +0000 UTC m=+1124.176129093" Sep 15 20:55:29.707885 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.707862 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:55:29.759226 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.759149 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw"] Sep 15 20:55:29.759602 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.759588 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" Sep 15 20:55:29.759652 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.759605 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" Sep 15 20:55:29.759652 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.759615 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" Sep 15 20:55:29.759652 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.759620 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" Sep 15 20:55:29.759741 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.759691 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" Sep 15 20:55:29.759741 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.759702 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerName="istio-proxy" Sep 15 20:55:29.762593 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.762576 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:29.764724 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.764705 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0143258d-dockercfg-2jc95\"" Sep 15 20:55:29.764842 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.764767 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0143258d\"" Sep 15 20:55:29.772993 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.772970 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw"] Sep 15 20:55:29.833586 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.833553 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-token\") pod \"c0209f6d-4426-4d66-8657-7885ccfbf026\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " Sep 15 20:55:29.833769 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.833613 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-socket\") pod \"c0209f6d-4426-4d66-8657-7885ccfbf026\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " Sep 15 20:55:29.833769 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.833643 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0209f6d-4426-4d66-8657-7885ccfbf026-istiod-ca-cert\") pod \"c0209f6d-4426-4d66-8657-7885ccfbf026\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " Sep 15 20:55:29.833769 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.833676 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-certs\") pod \"c0209f6d-4426-4d66-8657-7885ccfbf026\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " Sep 15 20:55:29.833769 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.833758 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-envoy\") pod \"c0209f6d-4426-4d66-8657-7885ccfbf026\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " Sep 15 20:55:29.833992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.833792 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l998\" (UniqueName: \"kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-kube-api-access-4l998\") pod \"c0209f6d-4426-4d66-8657-7885ccfbf026\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " Sep 15 20:55:29.833992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.833839 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-data\") pod \"c0209f6d-4426-4d66-8657-7885ccfbf026\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " Sep 15 20:55:29.833992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.833899 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-podinfo\") pod \"c0209f6d-4426-4d66-8657-7885ccfbf026\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " Sep 15 20:55:29.833992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.833930 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-credential-socket\") pod \"c0209f6d-4426-4d66-8657-7885ccfbf026\" (UID: \"c0209f6d-4426-4d66-8657-7885ccfbf026\") " Sep 15 20:55:29.833992 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.833976 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0209f6d-4426-4d66-8657-7885ccfbf026" (UID: "c0209f6d-4426-4d66-8657-7885ccfbf026"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:55:29.834241 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.834064 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0209f6d-4426-4d66-8657-7885ccfbf026" (UID: "c0209f6d-4426-4d66-8657-7885ccfbf026"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:55:29.834241 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.834070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2pjq\" (UniqueName: \"kubernetes.io/projected/e59c3a34-8932-4552-9bad-4be1d39ec54a-kube-api-access-v2pjq\") pod \"payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw\" (UID: \"e59c3a34-8932-4552-9bad-4be1d39ec54a\") " pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:29.834241 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.834093 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0209f6d-4426-4d66-8657-7885ccfbf026-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0209f6d-4426-4d66-8657-7885ccfbf026" (UID: "c0209f6d-4426-4d66-8657-7885ccfbf026"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:55:29.834241 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.834155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59c3a34-8932-4552-9bad-4be1d39ec54a-plugins-config-volume\") pod \"payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw\" (UID: \"e59c3a34-8932-4552-9bad-4be1d39ec54a\") " pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:29.834432 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.834293 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0209f6d-4426-4d66-8657-7885ccfbf026" (UID: "c0209f6d-4426-4d66-8657-7885ccfbf026"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:55:29.834432 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.834407 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:29.834432 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.834425 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0209f6d-4426-4d66-8657-7885ccfbf026-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:29.834615 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.834439 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:29.834615 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.834451 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:29.834615 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.834610 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0209f6d-4426-4d66-8657-7885ccfbf026" (UID: "c0209f6d-4426-4d66-8657-7885ccfbf026"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:55:29.836420 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.836389 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0209f6d-4426-4d66-8657-7885ccfbf026" (UID: "c0209f6d-4426-4d66-8657-7885ccfbf026"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:55:29.836582 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.836559 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0209f6d-4426-4d66-8657-7885ccfbf026" (UID: "c0209f6d-4426-4d66-8657-7885ccfbf026"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:55:29.836582 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.836572 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-kube-api-access-4l998" (OuterVolumeSpecName: "kube-api-access-4l998") pod "c0209f6d-4426-4d66-8657-7885ccfbf026" (UID: "c0209f6d-4426-4d66-8657-7885ccfbf026"). InnerVolumeSpecName "kube-api-access-4l998". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:55:29.836708 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.836666 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0209f6d-4426-4d66-8657-7885ccfbf026" (UID: "c0209f6d-4426-4d66-8657-7885ccfbf026"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 20:55:29.935670 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.935636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59c3a34-8932-4552-9bad-4be1d39ec54a-plugins-config-volume\") pod \"payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw\" (UID: \"e59c3a34-8932-4552-9bad-4be1d39ec54a\") " pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:29.935872 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.935779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pjq\" (UniqueName: \"kubernetes.io/projected/e59c3a34-8932-4552-9bad-4be1d39ec54a-kube-api-access-v2pjq\") pod \"payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw\" (UID: \"e59c3a34-8932-4552-9bad-4be1d39ec54a\") " pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:29.935872 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.935838 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:29.935872 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.935852 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l998\" (UniqueName: \"kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-kube-api-access-4l998\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:29.935872 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.935865 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:29.936028 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.935877 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0209f6d-4426-4d66-8657-7885ccfbf026-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:29.936028 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.935889 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0209f6d-4426-4d66-8657-7885ccfbf026-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:29.936273 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.936252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59c3a34-8932-4552-9bad-4be1d39ec54a-plugins-config-volume\") pod \"payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw\" (UID: \"e59c3a34-8932-4552-9bad-4be1d39ec54a\") " pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:29.943213 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.943180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pjq\" (UniqueName: \"kubernetes.io/projected/e59c3a34-8932-4552-9bad-4be1d39ec54a-kube-api-access-v2pjq\") pod \"payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw\" (UID: \"e59c3a34-8932-4552-9bad-4be1d39ec54a\") " pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:29.988537 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.988488 2579 generic.go:356] "Generic (PLEG): container finished" podID="c0209f6d-4426-4d66-8657-7885ccfbf026" containerID="a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8" exitCode=0 Sep 15 20:55:29.988698 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.988534 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" event={"ID":"c0209f6d-4426-4d66-8657-7885ccfbf026","Type":"ContainerDied","Data":"a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8"} Sep 15 20:55:29.988698 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.988572 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" Sep 15 20:55:29.988698 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.988633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf" event={"ID":"c0209f6d-4426-4d66-8657-7885ccfbf026","Type":"ContainerDied","Data":"93bee18f25254d6bc0fd327b6e8c3000af05c5cb2c9dbad5e295095197fe8421"} Sep 15 20:55:29.988698 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.988666 2579 scope.go:117] "RemoveContainer" containerID="a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8" Sep 15 20:55:29.999356 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:29.999333 2579 scope.go:117] "RemoveContainer" containerID="10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5" Sep 15 20:55:30.007409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.007390 2579 scope.go:117] "RemoveContainer" containerID="a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8" Sep 15 20:55:30.007718 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:55:30.007697 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8\": container with ID starting with a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8 not found: ID does not exist" containerID="a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8" Sep 15 20:55:30.007804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.007729 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8"} err="failed to get container status \"a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8\": rpc error: code = NotFound desc = could not find container \"a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8\": container with ID starting with a39f7d466cf55c44e64240f84a112521faa8194689bc6d3e2e1c9732b66307f8 not found: ID does not exist" Sep 15 20:55:30.007804 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.007754 2579 scope.go:117] "RemoveContainer" containerID="10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5" Sep 15 20:55:30.008045 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:55:30.008025 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5\": container with ID starting with 10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5 not found: ID does not exist" containerID="10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5" Sep 15 20:55:30.008084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.008055 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5"} err="failed to get container status \"10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5\": rpc error: code = NotFound desc = could not find container \"10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5\": container with ID starting with 10811ec9cfdd02666031788dd7ab6f018aac57a9496b1208c5ca0222c79cb9c5 not found: ID does not exist" Sep 15 20:55:30.014946 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.014924 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf"] Sep 15 20:55:30.025119 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.025093 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e8af3fa9-gw-openshift-default-87bf6785b-tnqrf"] Sep 15 20:55:30.073258 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.073224 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:30.347723 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.347700 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw"] Sep 15 20:55:30.349698 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:55:30.349668 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59c3a34_8932_4552_9bad_4be1d39ec54a.slice/crio-7de73ef0763165feaa948efefa397a60469e216d9ec50258ae43811def2e19d6 WatchSource:0}: Error finding container 7de73ef0763165feaa948efefa397a60469e216d9ec50258ae43811def2e19d6: Status 404 returned error can't find the container with id 7de73ef0763165feaa948efefa397a60469e216d9ec50258ae43811def2e19d6 Sep 15 20:55:30.996841 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.996811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" event={"ID":"e59c3a34-8932-4552-9bad-4be1d39ec54a","Type":"ContainerStarted","Data":"a28db8714832006ed6621f974431c4440e4f660d1cf674be2150721c82ad328f"} Sep 15 20:55:30.997225 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.996898 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" event={"ID":"e59c3a34-8932-4552-9bad-4be1d39ec54a","Type":"ContainerStarted","Data":"7de73ef0763165feaa948efefa397a60469e216d9ec50258ae43811def2e19d6"} Sep 15 20:55:30.997225 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:30.996951 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:31.012254 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:31.012208 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" podStartSLOduration=2.012193594 podStartE2EDuration="2.012193594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:55:31.010354476 +0000 UTC m=+1128.204307932" watchObservedRunningTime="2026-09-15 20:55:31.012193594 +0000 UTC m=+1128.206147047" Sep 15 20:55:31.311789 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:31.311758 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0209f6d-4426-4d66-8657-7885ccfbf026" path="/var/lib/kubelet/pods/c0209f6d-4426-4d66-8657-7885ccfbf026/volumes" Sep 15 20:55:36.470585 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:36.470542 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:36.471028 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:36.470619 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:36.488176 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:36.488150 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:37.038475 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:37.038450 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:55:42.003335 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:42.003304 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:43.005669 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:43.005633 2579 patch_prober.go:28] interesting pod/e2e-ait-0143258d-openshift-default-df7988478-d2tg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 15 20:55:43.006064 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:43.005713 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 15 20:55:44.065621 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:44.065590 2579 generic.go:356] "Generic (PLEG): container finished" podID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerID="9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b" exitCode=0 Sep 15 20:55:44.066073 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:44.065667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" event={"ID":"9ad0394b-adea-4d97-a372-afb5d51ffaec","Type":"ContainerDied","Data":"9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b"} Sep 15 20:55:44.066073 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:44.066059 2579 scope.go:117] "RemoveContainer" containerID="9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b" Sep 15 20:55:44.067208 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:44.067178 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:55:44.067307 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:44.067246 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:55:44.067307 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:44.067278 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:55:45.072114 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:45.072079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" event={"ID":"9ad0394b-adea-4d97-a372-afb5d51ffaec","Type":"ContainerStarted","Data":"2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7"} Sep 15 20:55:45.447179 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:45.447088 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:45.447379 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:45.447357 2579 patch_prober.go:28] interesting pod/e2e-ait-0143258d-openshift-default-df7988478-d2tg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:55:45.447533 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:45.447480 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 15 20:55:46.447294 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:46.447248 2579 patch_prober.go:28] interesting pod/e2e-ait-0143258d-openshift-default-df7988478-d2tg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:55:46.447850 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:46.447313 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 15 20:55:47.447737 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:47.447695 2579 patch_prober.go:28] interesting pod/e2e-ait-0143258d-openshift-default-df7988478-d2tg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:55:47.448102 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:47.447761 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 15 20:55:48.447029 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:48.446990 2579 patch_prober.go:28] interesting pod/e2e-ait-0143258d-openshift-default-df7988478-d2tg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:55:48.447218 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:48.447050 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 15 20:55:49.447170 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:49.447130 2579 patch_prober.go:28] interesting pod/e2e-ait-0143258d-openshift-default-df7988478-d2tg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:55:49.447636 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:49.447185 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 15 20:55:50.492051 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:50.492021 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:50.492428 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:50.492293 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:50.493135 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:50.493115 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:55:53.312553 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:53.312522 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d"] Sep 15 20:55:53.316250 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:53.316229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d" Sep 15 20:55:53.318820 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:53.318797 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 20:55:53.320153 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:53.320131 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d"] Sep 15 20:55:53.363852 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:53.363825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1c8a60-3f69-435c-b611-9763b50fa655-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0143258d-5zw9d\" (UID: \"2c1c8a60-3f69-435c-b611-9763b50fa655\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d" Sep 15 20:55:53.465036 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:53.465006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1c8a60-3f69-435c-b611-9763b50fa655-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0143258d-5zw9d\" (UID: \"2c1c8a60-3f69-435c-b611-9763b50fa655\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d" Sep 15 20:55:53.465666 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:53.465649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1c8a60-3f69-435c-b611-9763b50fa655-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0143258d-5zw9d\" (UID: \"2c1c8a60-3f69-435c-b611-9763b50fa655\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d" Sep 15 20:55:53.628772 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:53.628686 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d" Sep 15 20:55:53.930611 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:53.930584 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d"] Sep 15 20:55:53.933049 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:55:53.933023 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1c8a60_3f69_435c_b611_9763b50fa655.slice/crio-bdb098ae2e28822109f5431bdbfd443ab4a61aa86e4b586f3eb9b23bd1fc9d69 WatchSource:0}: Error finding container bdb098ae2e28822109f5431bdbfd443ab4a61aa86e4b586f3eb9b23bd1fc9d69: Status 404 returned error can't find the container with id bdb098ae2e28822109f5431bdbfd443ab4a61aa86e4b586f3eb9b23bd1fc9d69 Sep 15 20:55:54.130058 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:54.130019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d" event={"ID":"2c1c8a60-3f69-435c-b611-9763b50fa655","Type":"ContainerStarted","Data":"bdb098ae2e28822109f5431bdbfd443ab4a61aa86e4b586f3eb9b23bd1fc9d69"} Sep 15 20:55:55.136302 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:55.136270 2579 generic.go:356] "Generic (PLEG): container finished" podID="2c1c8a60-3f69-435c-b611-9763b50fa655" containerID="d0b0547aae8ec8cca5ace148ed9a14e33eb6748674a2e6e264216384f7592292" exitCode=0 Sep 15 20:55:55.136810 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:55.136363 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d" event={"ID":"2c1c8a60-3f69-435c-b611-9763b50fa655","Type":"ContainerDied","Data":"d0b0547aae8ec8cca5ace148ed9a14e33eb6748674a2e6e264216384f7592292"} Sep 15 20:55:56.277142 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:56.277117 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d" Sep 15 20:55:56.293317 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:56.293097 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1c8a60-3f69-435c-b611-9763b50fa655-maas-api-service-ca\") pod \"2c1c8a60-3f69-435c-b611-9763b50fa655\" (UID: \"2c1c8a60-3f69-435c-b611-9763b50fa655\") " Sep 15 20:55:56.293594 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:56.293563 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1c8a60-3f69-435c-b611-9763b50fa655-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c1c8a60-3f69-435c-b611-9763b50fa655" (UID: "2c1c8a60-3f69-435c-b611-9763b50fa655"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:55:56.394066 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:56.393980 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1c8a60-3f69-435c-b611-9763b50fa655-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:57.147169 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:57.147137 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d" event={"ID":"2c1c8a60-3f69-435c-b611-9763b50fa655","Type":"ContainerDied","Data":"bdb098ae2e28822109f5431bdbfd443ab4a61aa86e4b586f3eb9b23bd1fc9d69"} Sep 15 20:55:57.147169 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:57.147161 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d" Sep 15 20:55:57.147398 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:57.147183 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb098ae2e28822109f5431bdbfd443ab4a61aa86e4b586f3eb9b23bd1fc9d69" Sep 15 20:55:59.027529 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.027479 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw"] Sep 15 20:55:59.028028 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.027720 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" podUID="e59c3a34-8932-4552-9bad-4be1d39ec54a" containerName="payload-processing" containerID="cri-o://a28db8714832006ed6621f974431c4440e4f660d1cf674be2150721c82ad328f" gracePeriod=30 Sep 15 20:55:59.159121 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.159094 2579 generic.go:356] "Generic (PLEG): container finished" podID="e59c3a34-8932-4552-9bad-4be1d39ec54a" containerID="a28db8714832006ed6621f974431c4440e4f660d1cf674be2150721c82ad328f" exitCode=0 Sep 15 20:55:59.159266 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.159174 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" event={"ID":"e59c3a34-8932-4552-9bad-4be1d39ec54a","Type":"ContainerDied","Data":"a28db8714832006ed6621f974431c4440e4f660d1cf674be2150721c82ad328f"} Sep 15 20:55:59.177257 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.177223 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d"] Sep 15 20:55:59.179117 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.179091 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0143258d-5zw9d"] Sep 15 20:55:59.290769 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.290709 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:55:59.312916 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.312887 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c1c8a60-3f69-435c-b611-9763b50fa655" path="/var/lib/kubelet/pods/2c1c8a60-3f69-435c-b611-9763b50fa655/volumes" Sep 15 20:55:59.323423 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.323399 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59c3a34-8932-4552-9bad-4be1d39ec54a-plugins-config-volume\") pod \"e59c3a34-8932-4552-9bad-4be1d39ec54a\" (UID: \"e59c3a34-8932-4552-9bad-4be1d39ec54a\") " Sep 15 20:55:59.323619 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.323435 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2pjq\" (UniqueName: \"kubernetes.io/projected/e59c3a34-8932-4552-9bad-4be1d39ec54a-kube-api-access-v2pjq\") pod \"e59c3a34-8932-4552-9bad-4be1d39ec54a\" (UID: \"e59c3a34-8932-4552-9bad-4be1d39ec54a\") " Sep 15 20:55:59.323767 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.323742 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59c3a34-8932-4552-9bad-4be1d39ec54a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e59c3a34-8932-4552-9bad-4be1d39ec54a" (UID: "e59c3a34-8932-4552-9bad-4be1d39ec54a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:55:59.325882 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.325853 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59c3a34-8932-4552-9bad-4be1d39ec54a-kube-api-access-v2pjq" (OuterVolumeSpecName: "kube-api-access-v2pjq") pod "e59c3a34-8932-4552-9bad-4be1d39ec54a" (UID: "e59c3a34-8932-4552-9bad-4be1d39ec54a"). InnerVolumeSpecName "kube-api-access-v2pjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:55:59.425022 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.424970 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59c3a34-8932-4552-9bad-4be1d39ec54a-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:55:59.425022 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:55:59.425009 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2pjq\" (UniqueName: \"kubernetes.io/projected/e59c3a34-8932-4552-9bad-4be1d39ec54a-kube-api-access-v2pjq\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:00.166055 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:00.166022 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" event={"ID":"e59c3a34-8932-4552-9bad-4be1d39ec54a","Type":"ContainerDied","Data":"7de73ef0763165feaa948efefa397a60469e216d9ec50258ae43811def2e19d6"} Sep 15 20:56:00.166486 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:00.166078 2579 scope.go:117] "RemoveContainer" containerID="a28db8714832006ed6621f974431c4440e4f660d1cf674be2150721c82ad328f" Sep 15 20:56:00.166486 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:00.166036 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw" Sep 15 20:56:00.186175 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:00.186148 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw"] Sep 15 20:56:00.192064 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:00.192041 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0143258d-544bd9866f-hvlgw"] Sep 15 20:56:00.474181 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:00.474105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9"] Sep 15 20:56:00.474369 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:00.474332 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" containerID="cri-o://2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7" gracePeriod=30 Sep 15 20:56:01.303912 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.303873 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g"] Sep 15 20:56:01.304390 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.304369 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1c8a60-3f69-435c-b611-9763b50fa655" containerName="revoke-keys" Sep 15 20:56:01.304471 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.304395 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1c8a60-3f69-435c-b611-9763b50fa655" containerName="revoke-keys" Sep 15 20:56:01.304471 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.304417 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e59c3a34-8932-4552-9bad-4be1d39ec54a" containerName="payload-processing" Sep 15 20:56:01.304471 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.304427 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e59c3a34-8932-4552-9bad-4be1d39ec54a" containerName="payload-processing" Sep 15 20:56:01.304623 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.304558 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e59c3a34-8932-4552-9bad-4be1d39ec54a" containerName="payload-processing" Sep 15 20:56:01.304623 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.304576 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1c8a60-3f69-435c-b611-9763b50fa655" containerName="revoke-keys" Sep 15 20:56:01.309284 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.309260 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.311913 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.311889 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a7948686-openshift-default-dockercfg-676pc\"" Sep 15 20:56:01.318532 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.317855 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e59c3a34-8932-4552-9bad-4be1d39ec54a" path="/var/lib/kubelet/pods/e59c3a34-8932-4552-9bad-4be1d39ec54a/volumes" Sep 15 20:56:01.325054 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.325026 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g"] Sep 15 20:56:01.343918 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.343888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-socket\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.344100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.343922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-certs\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.344100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.343965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-data\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.344100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.343990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-token\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.344100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.344020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istiod-ca-cert\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.344100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.344046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-credential-socket\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.344100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.344069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-envoy\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.344401 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.344157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-podinfo\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.344401 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.344216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhrj4\" (UniqueName: \"kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-kube-api-access-qhrj4\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-socket\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-certs\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445334 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-data\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445334 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-token\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445334 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istiod-ca-cert\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445334 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-credential-socket\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445334 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-envoy\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445334 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-podinfo\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445688 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhrj4\" (UniqueName: \"kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-kube-api-access-qhrj4\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445688 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445565 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-socket\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445688 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445584 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-certs\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445688 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-credential-socket\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445901 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-data\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.445959 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.445927 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istiod-ca-cert\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.447703 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.447678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-envoy\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.448096 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.448077 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-podinfo\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.453029 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.453008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhrj4\" (UniqueName: \"kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-kube-api-access-qhrj4\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.453282 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.453262 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-token\") pod \"e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.623577 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.623462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:01.904061 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.904035 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g"] Sep 15 20:56:01.906324 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:56:01.906281 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180ed213_5e9a_4eab_ac16_2cb3fae30eec.slice/crio-f601963b99778ed19df5ac9e04b8558cf3e99a105d52966fc0e7d8809fb8b1b2 WatchSource:0}: Error finding container f601963b99778ed19df5ac9e04b8558cf3e99a105d52966fc0e7d8809fb8b1b2: Status 404 returned error can't find the container with id f601963b99778ed19df5ac9e04b8558cf3e99a105d52966fc0e7d8809fb8b1b2 Sep 15 20:56:01.908742 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.908701 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:56:01.908838 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.908817 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:56:01.908879 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:01.908868 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:56:02.182898 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:02.182816 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" event={"ID":"180ed213-5e9a-4eab-ac16-2cb3fae30eec","Type":"ContainerStarted","Data":"d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9"} Sep 15 20:56:02.182898 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:02.182894 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" event={"ID":"180ed213-5e9a-4eab-ac16-2cb3fae30eec","Type":"ContainerStarted","Data":"f601963b99778ed19df5ac9e04b8558cf3e99a105d52966fc0e7d8809fb8b1b2"} Sep 15 20:56:02.204148 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:02.204099 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" podStartSLOduration=1.20408436 podStartE2EDuration="1.20408436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:56:02.201228546 +0000 UTC m=+1159.395182002" watchObservedRunningTime="2026-09-15 20:56:02.20408436 +0000 UTC m=+1159.398037815" Sep 15 20:56:02.624571 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:02.624538 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:02.629686 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:02.629664 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:02.843552 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:02.843516 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6"] Sep 15 20:56:02.847256 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:02.847237 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6" Sep 15 20:56:02.849684 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:02.849653 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 20:56:02.854686 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:02.854654 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6"] Sep 15 20:56:02.960548 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:02.960443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a2e2dc-4ae0-4935-9d44-dbf190d9e20f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6\" (UID: \"14a2e2dc-4ae0-4935-9d44-dbf190d9e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6" Sep 15 20:56:03.061846 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:03.061808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a2e2dc-4ae0-4935-9d44-dbf190d9e20f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6\" (UID: \"14a2e2dc-4ae0-4935-9d44-dbf190d9e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6" Sep 15 20:56:03.062486 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:03.062466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a2e2dc-4ae0-4935-9d44-dbf190d9e20f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6\" (UID: \"14a2e2dc-4ae0-4935-9d44-dbf190d9e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6" Sep 15 20:56:03.159790 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:03.159744 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6" Sep 15 20:56:03.188316 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:03.188283 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:03.189243 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:03.189221 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:03.432013 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:03.431984 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6"] Sep 15 20:56:03.435680 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:56:03.435654 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a2e2dc_4ae0_4935_9d44_dbf190d9e20f.slice/crio-1044a8902573fc121828edb96a374e7a764c8d1a90c4930d80c5c959d8f193d9 WatchSource:0}: Error finding container 1044a8902573fc121828edb96a374e7a764c8d1a90c4930d80c5c959d8f193d9: Status 404 returned error can't find the container with id 1044a8902573fc121828edb96a374e7a764c8d1a90c4930d80c5c959d8f193d9 Sep 15 20:56:04.194704 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:04.194672 2579 generic.go:356] "Generic (PLEG): container finished" podID="14a2e2dc-4ae0-4935-9d44-dbf190d9e20f" containerID="ffc751baeb6917f73283e8b19c068baeb070ed39bc4c40725846ef22aadd29a8" exitCode=0 Sep 15 20:56:04.195132 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:04.194762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6" event={"ID":"14a2e2dc-4ae0-4935-9d44-dbf190d9e20f","Type":"ContainerDied","Data":"ffc751baeb6917f73283e8b19c068baeb070ed39bc4c40725846ef22aadd29a8"} Sep 15 20:56:04.195132 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:04.194820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6" event={"ID":"14a2e2dc-4ae0-4935-9d44-dbf190d9e20f","Type":"ContainerStarted","Data":"1044a8902573fc121828edb96a374e7a764c8d1a90c4930d80c5c959d8f193d9"} Sep 15 20:56:05.334695 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.334672 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6" Sep 15 20:56:05.385629 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.385596 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a2e2dc-4ae0-4935-9d44-dbf190d9e20f-maas-api-service-ca\") pod \"14a2e2dc-4ae0-4935-9d44-dbf190d9e20f\" (UID: \"14a2e2dc-4ae0-4935-9d44-dbf190d9e20f\") " Sep 15 20:56:05.385962 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.385927 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a2e2dc-4ae0-4935-9d44-dbf190d9e20f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14a2e2dc-4ae0-4935-9d44-dbf190d9e20f" (UID: "14a2e2dc-4ae0-4935-9d44-dbf190d9e20f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:56:05.486577 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.486553 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a2e2dc-4ae0-4935-9d44-dbf190d9e20f-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:05.493933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.493905 2579 patch_prober.go:28] interesting pod/e2e-ait-0143258d-openshift-default-df7988478-d2tg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:56:05.494045 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.493954 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 15 20:56:05.726673 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.726649 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:56:05.789226 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789195 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-certs\") pod \"9ad0394b-adea-4d97-a372-afb5d51ffaec\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " Sep 15 20:56:05.789395 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789268 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djw29\" (UniqueName: \"kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-kube-api-access-djw29\") pod \"9ad0394b-adea-4d97-a372-afb5d51ffaec\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " Sep 15 20:56:05.789395 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789289 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-podinfo\") pod \"9ad0394b-adea-4d97-a372-afb5d51ffaec\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " Sep 15 20:56:05.789395 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789305 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad0394b-adea-4d97-a372-afb5d51ffaec-istiod-ca-cert\") pod \"9ad0394b-adea-4d97-a372-afb5d51ffaec\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " Sep 15 20:56:05.789395 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789342 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-credential-socket\") pod \"9ad0394b-adea-4d97-a372-afb5d51ffaec\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " Sep 15 20:56:05.789637 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789420 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-token\") pod \"9ad0394b-adea-4d97-a372-afb5d51ffaec\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " Sep 15 20:56:05.789637 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789470 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-socket\") pod \"9ad0394b-adea-4d97-a372-afb5d51ffaec\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " Sep 15 20:56:05.789637 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789492 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ad0394b-adea-4d97-a372-afb5d51ffaec" (UID: "9ad0394b-adea-4d97-a372-afb5d51ffaec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:05.789637 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789535 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-data\") pod \"9ad0394b-adea-4d97-a372-afb5d51ffaec\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " Sep 15 20:56:05.789637 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789610 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-envoy\") pod \"9ad0394b-adea-4d97-a372-afb5d51ffaec\" (UID: \"9ad0394b-adea-4d97-a372-afb5d51ffaec\") " Sep 15 20:56:05.789637 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789632 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ad0394b-adea-4d97-a372-afb5d51ffaec" (UID: "9ad0394b-adea-4d97-a372-afb5d51ffaec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:05.789921 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789743 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ad0394b-adea-4d97-a372-afb5d51ffaec" (UID: "9ad0394b-adea-4d97-a372-afb5d51ffaec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:05.789921 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.789783 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ad0394b-adea-4d97-a372-afb5d51ffaec" (UID: "9ad0394b-adea-4d97-a372-afb5d51ffaec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:05.790206 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.790050 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad0394b-adea-4d97-a372-afb5d51ffaec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ad0394b-adea-4d97-a372-afb5d51ffaec" (UID: "9ad0394b-adea-4d97-a372-afb5d51ffaec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:56:05.790269 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.790204 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:05.790269 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.790226 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:05.790269 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.790238 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:05.790269 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.790251 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:05.791590 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.791566 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ad0394b-adea-4d97-a372-afb5d51ffaec" (UID: "9ad0394b-adea-4d97-a372-afb5d51ffaec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 20:56:05.791590 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.791579 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-kube-api-access-djw29" (OuterVolumeSpecName: "kube-api-access-djw29") pod "9ad0394b-adea-4d97-a372-afb5d51ffaec" (UID: "9ad0394b-adea-4d97-a372-afb5d51ffaec"). InnerVolumeSpecName "kube-api-access-djw29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:56:05.791723 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.791588 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ad0394b-adea-4d97-a372-afb5d51ffaec" (UID: "9ad0394b-adea-4d97-a372-afb5d51ffaec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:56:05.791723 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.791688 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ad0394b-adea-4d97-a372-afb5d51ffaec" (UID: "9ad0394b-adea-4d97-a372-afb5d51ffaec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:05.890730 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.890702 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djw29\" (UniqueName: \"kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-kube-api-access-djw29\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:05.890730 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.890725 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:05.890730 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.890734 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad0394b-adea-4d97-a372-afb5d51ffaec-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:05.890944 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.890742 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:05.890944 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:05.890751 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad0394b-adea-4d97-a372-afb5d51ffaec-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:06.213719 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.213637 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6" Sep 15 20:56:06.213719 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.213637 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6" event={"ID":"14a2e2dc-4ae0-4935-9d44-dbf190d9e20f","Type":"ContainerDied","Data":"1044a8902573fc121828edb96a374e7a764c8d1a90c4930d80c5c959d8f193d9"} Sep 15 20:56:06.213950 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.213727 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1044a8902573fc121828edb96a374e7a764c8d1a90c4930d80c5c959d8f193d9" Sep 15 20:56:06.215702 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.215676 2579 generic.go:356] "Generic (PLEG): container finished" podID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerID="2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7" exitCode=0 Sep 15 20:56:06.215850 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.215724 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" Sep 15 20:56:06.215850 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.215727 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" event={"ID":"9ad0394b-adea-4d97-a372-afb5d51ffaec","Type":"ContainerDied","Data":"2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7"} Sep 15 20:56:06.215850 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.215769 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9" event={"ID":"9ad0394b-adea-4d97-a372-afb5d51ffaec","Type":"ContainerDied","Data":"3d244b454027337c5846037974acf519df6d299d48220c7c6681dd2b44b75fd8"} Sep 15 20:56:06.215850 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.215796 2579 scope.go:117] "RemoveContainer" containerID="2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7" Sep 15 20:56:06.225240 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.225198 2579 scope.go:117] "RemoveContainer" containerID="9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b" Sep 15 20:56:06.233430 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.233410 2579 scope.go:117] "RemoveContainer" containerID="2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7" Sep 15 20:56:06.233697 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:56:06.233678 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7\": container with ID starting with 2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7 not found: ID does not exist" containerID="2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7" Sep 15 20:56:06.233755 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.233705 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7"} err="failed to get container status \"2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7\": rpc error: code = NotFound desc = could not find container \"2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7\": container with ID starting with 2a4d6bda786f0e6c65c14432309a2b59582e92be86311414f9a96f5d7aea03f7 not found: ID does not exist" Sep 15 20:56:06.233755 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.233721 2579 scope.go:117] "RemoveContainer" containerID="9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b" Sep 15 20:56:06.233984 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:56:06.233969 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b\": container with ID starting with 9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b not found: ID does not exist" containerID="9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b" Sep 15 20:56:06.234033 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.233985 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b"} err="failed to get container status \"9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b\": rpc error: code = NotFound desc = could not find container \"9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b\": container with ID starting with 9cffbf261b6d6eaf8bf553a1b56b223dd429129ace9c5caeb0759c47eed6931b not found: ID does not exist" Sep 15 20:56:06.238345 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.238321 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9"] Sep 15 20:56:06.241584 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:06.241566 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0143258d-openshift-default-df7988478-d2tg9"] Sep 15 20:56:07.313112 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:07.313082 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" path="/var/lib/kubelet/pods/9ad0394b-adea-4d97-a372-afb5d51ffaec/volumes" Sep 15 20:56:09.130623 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.130590 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt"] Sep 15 20:56:09.131009 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.130978 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a2e2dc-4ae0-4935-9d44-dbf190d9e20f" containerName="revoke-keys" Sep 15 20:56:09.131009 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.130991 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a2e2dc-4ae0-4935-9d44-dbf190d9e20f" containerName="revoke-keys" Sep 15 20:56:09.131009 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.131008 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" Sep 15 20:56:09.131106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.131014 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" Sep 15 20:56:09.131106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.131023 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" Sep 15 20:56:09.131106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.131029 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" Sep 15 20:56:09.131106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.131084 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a2e2dc-4ae0-4935-9d44-dbf190d9e20f" containerName="revoke-keys" Sep 15 20:56:09.131106 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.131094 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" Sep 15 20:56:09.135565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.135548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:09.137739 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.137719 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a7948686\"" Sep 15 20:56:09.137844 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.137720 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a7948686-dockercfg-2tvtv\"" Sep 15 20:56:09.144101 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.144080 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt"] Sep 15 20:56:09.222380 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.222346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7qh\" (UniqueName: \"kubernetes.io/projected/64fd895a-ebb4-4a96-b660-a00f4094620a-kube-api-access-pd7qh\") pod \"payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt\" (UID: \"64fd895a-ebb4-4a96-b660-a00f4094620a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:09.222553 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.222426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fd895a-ebb4-4a96-b660-a00f4094620a-plugins-config-volume\") pod \"payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt\" (UID: \"64fd895a-ebb4-4a96-b660-a00f4094620a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:09.323590 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.323551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fd895a-ebb4-4a96-b660-a00f4094620a-plugins-config-volume\") pod \"payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt\" (UID: \"64fd895a-ebb4-4a96-b660-a00f4094620a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:09.323797 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.323675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7qh\" (UniqueName: \"kubernetes.io/projected/64fd895a-ebb4-4a96-b660-a00f4094620a-kube-api-access-pd7qh\") pod \"payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt\" (UID: \"64fd895a-ebb4-4a96-b660-a00f4094620a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:09.324247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.324228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fd895a-ebb4-4a96-b660-a00f4094620a-plugins-config-volume\") pod \"payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt\" (UID: \"64fd895a-ebb4-4a96-b660-a00f4094620a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:09.333510 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.333480 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7qh\" (UniqueName: \"kubernetes.io/projected/64fd895a-ebb4-4a96-b660-a00f4094620a-kube-api-access-pd7qh\") pod \"payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt\" (UID: \"64fd895a-ebb4-4a96-b660-a00f4094620a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:09.446522 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.446421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:09.719450 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:09.719421 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt"] Sep 15 20:56:09.721837 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:56:09.721805 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fd895a_ebb4_4a96_b660_a00f4094620a.slice/crio-462d5746ae6e2bac20c8717ade91dd26fdafb234fb9f68d897523b55cf54a14c WatchSource:0}: Error finding container 462d5746ae6e2bac20c8717ade91dd26fdafb234fb9f68d897523b55cf54a14c: Status 404 returned error can't find the container with id 462d5746ae6e2bac20c8717ade91dd26fdafb234fb9f68d897523b55cf54a14c Sep 15 20:56:10.238408 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:10.238368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" event={"ID":"64fd895a-ebb4-4a96-b660-a00f4094620a","Type":"ContainerStarted","Data":"91937c2f36c752c63f681af167624d146f6191d03be1ffd4b6b319a9994b92c2"} Sep 15 20:56:10.238891 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:10.238450 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" event={"ID":"64fd895a-ebb4-4a96-b660-a00f4094620a","Type":"ContainerStarted","Data":"462d5746ae6e2bac20c8717ade91dd26fdafb234fb9f68d897523b55cf54a14c"} Sep 15 20:56:10.238891 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:10.238474 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:10.259043 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:10.258992 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" podStartSLOduration=1.258971834 podStartE2EDuration="1.258971834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:56:10.257158786 +0000 UTC m=+1167.451112256" watchObservedRunningTime="2026-09-15 20:56:10.258971834 +0000 UTC m=+1167.452925291" Sep 15 20:56:13.537792 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.537761 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f"] Sep 15 20:56:13.538263 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.538046 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" podUID="7463aa39-ce69-4a41-ba13-4fb5f8960df4" containerName="payload-pre-processing" containerID="cri-o://f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d" gracePeriod=30 Sep 15 20:56:13.681138 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.681091 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6"] Sep 15 20:56:13.683124 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.683101 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b9ff68-p4cj6"] Sep 15 20:56:13.784159 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.784132 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:56:13.869241 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.869159 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7463aa39-ce69-4a41-ba13-4fb5f8960df4-plugins-config-volume\") pod \"7463aa39-ce69-4a41-ba13-4fb5f8960df4\" (UID: \"7463aa39-ce69-4a41-ba13-4fb5f8960df4\") " Sep 15 20:56:13.869386 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.869275 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77q7f\" (UniqueName: \"kubernetes.io/projected/7463aa39-ce69-4a41-ba13-4fb5f8960df4-kube-api-access-77q7f\") pod \"7463aa39-ce69-4a41-ba13-4fb5f8960df4\" (UID: \"7463aa39-ce69-4a41-ba13-4fb5f8960df4\") " Sep 15 20:56:13.869636 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.869567 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7463aa39-ce69-4a41-ba13-4fb5f8960df4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7463aa39-ce69-4a41-ba13-4fb5f8960df4" (UID: "7463aa39-ce69-4a41-ba13-4fb5f8960df4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:56:13.869688 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.869665 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7463aa39-ce69-4a41-ba13-4fb5f8960df4-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:13.871486 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.871465 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7463aa39-ce69-4a41-ba13-4fb5f8960df4-kube-api-access-77q7f" (OuterVolumeSpecName: "kube-api-access-77q7f") pod "7463aa39-ce69-4a41-ba13-4fb5f8960df4" (UID: "7463aa39-ce69-4a41-ba13-4fb5f8960df4"). InnerVolumeSpecName "kube-api-access-77q7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:56:13.970954 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:13.970919 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77q7f\" (UniqueName: \"kubernetes.io/projected/7463aa39-ce69-4a41-ba13-4fb5f8960df4-kube-api-access-77q7f\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:14.259817 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:14.259727 2579 generic.go:356] "Generic (PLEG): container finished" podID="7463aa39-ce69-4a41-ba13-4fb5f8960df4" containerID="f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d" exitCode=0 Sep 15 20:56:14.259817 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:14.259770 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" Sep 15 20:56:14.259817 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:14.259793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" event={"ID":"7463aa39-ce69-4a41-ba13-4fb5f8960df4","Type":"ContainerDied","Data":"f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d"} Sep 15 20:56:14.260039 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:14.259841 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f" event={"ID":"7463aa39-ce69-4a41-ba13-4fb5f8960df4","Type":"ContainerDied","Data":"7a734c3f4e4254e5c8b110d1bf47f5e9b747c2f10616ef0fb9f66bca612fc2e3"} Sep 15 20:56:14.260039 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:14.259867 2579 scope.go:117] "RemoveContainer" containerID="f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d" Sep 15 20:56:14.269010 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:14.268993 2579 scope.go:117] "RemoveContainer" containerID="f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d" Sep 15 20:56:14.269279 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:56:14.269260 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d\": container with ID starting with f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d not found: ID does not exist" containerID="f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d" Sep 15 20:56:14.269339 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:14.269289 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d"} err="failed to get container status \"f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d\": rpc error: code = NotFound desc = could not find container \"f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d\": container with ID starting with f7342fa75bfc33f078dc481efbdbe0773c48a7fad19de5159f23c33ba922660d not found: ID does not exist" Sep 15 20:56:14.281877 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:14.281813 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f"] Sep 15 20:56:14.284063 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:14.284043 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b9ff68-d44cc8566-g956f"] Sep 15 20:56:15.311690 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:15.311656 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a2e2dc-4ae0-4935-9d44-dbf190d9e20f" path="/var/lib/kubelet/pods/14a2e2dc-4ae0-4935-9d44-dbf190d9e20f/volumes" Sep 15 20:56:15.312093 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:15.312023 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7463aa39-ce69-4a41-ba13-4fb5f8960df4" path="/var/lib/kubelet/pods/7463aa39-ce69-4a41-ba13-4fb5f8960df4/volumes" Sep 15 20:56:18.220972 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.220936 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw"] Sep 15 20:56:18.221358 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.221319 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7463aa39-ce69-4a41-ba13-4fb5f8960df4" containerName="payload-pre-processing" Sep 15 20:56:18.221358 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.221331 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7463aa39-ce69-4a41-ba13-4fb5f8960df4" containerName="payload-pre-processing" Sep 15 20:56:18.221436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.221407 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7463aa39-ce69-4a41-ba13-4fb5f8960df4" containerName="payload-pre-processing" Sep 15 20:56:18.221436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.221416 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad0394b-adea-4d97-a372-afb5d51ffaec" containerName="istio-proxy" Sep 15 20:56:18.225867 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.225850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" Sep 15 20:56:18.228033 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.228011 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 20:56:18.231669 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.231643 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw"] Sep 15 20:56:18.315234 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.315200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65998f0-8e70-4f61-9811-fefe227299fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a7948686-rnwpw\" (UID: \"f65998f0-8e70-4f61-9811-fefe227299fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" Sep 15 20:56:18.416303 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.416260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65998f0-8e70-4f61-9811-fefe227299fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a7948686-rnwpw\" (UID: \"f65998f0-8e70-4f61-9811-fefe227299fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" Sep 15 20:56:18.416905 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.416885 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65998f0-8e70-4f61-9811-fefe227299fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a7948686-rnwpw\" (UID: \"f65998f0-8e70-4f61-9811-fefe227299fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" Sep 15 20:56:18.537471 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.537441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" Sep 15 20:56:18.805226 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:18.805151 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw"] Sep 15 20:56:18.808681 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:56:18.808659 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65998f0_8e70_4f61_9811_fefe227299fb.slice/crio-91fd132ecd7cb9d983fe333556a8ce5c81cb57374be73d4e09d48e1204de5d65 WatchSource:0}: Error finding container 91fd132ecd7cb9d983fe333556a8ce5c81cb57374be73d4e09d48e1204de5d65: Status 404 returned error can't find the container with id 91fd132ecd7cb9d983fe333556a8ce5c81cb57374be73d4e09d48e1204de5d65 Sep 15 20:56:19.118980 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:19.118880 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq"] Sep 15 20:56:19.291813 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:19.291766 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" event={"ID":"f65998f0-8e70-4f61-9811-fefe227299fb","Type":"ContainerStarted","Data":"0e24d0aefd446fba5d86d2be52602524e7802e32b0b32c6f5f1a8823d1f9dbe5"} Sep 15 20:56:19.292217 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:19.291882 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" event={"ID":"f65998f0-8e70-4f61-9811-fefe227299fb","Type":"ContainerStarted","Data":"91fd132ecd7cb9d983fe333556a8ce5c81cb57374be73d4e09d48e1204de5d65"} Sep 15 20:56:19.309749 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:19.309698 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" podStartSLOduration=1.309680451 podStartE2EDuration="1.309680451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:56:19.307674068 +0000 UTC m=+1176.501627524" watchObservedRunningTime="2026-09-15 20:56:19.309680451 +0000 UTC m=+1176.503633910" Sep 15 20:56:20.299536 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:20.299479 2579 generic.go:356] "Generic (PLEG): container finished" podID="f65998f0-8e70-4f61-9811-fefe227299fb" containerID="0e24d0aefd446fba5d86d2be52602524e7802e32b0b32c6f5f1a8823d1f9dbe5" exitCode=0 Sep 15 20:56:20.299920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:20.299534 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" event={"ID":"f65998f0-8e70-4f61-9811-fefe227299fb","Type":"ContainerDied","Data":"0e24d0aefd446fba5d86d2be52602524e7802e32b0b32c6f5f1a8823d1f9dbe5"} Sep 15 20:56:21.245379 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:21.245352 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:21.457030 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:21.457009 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" Sep 15 20:56:21.649340 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:21.649307 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65998f0-8e70-4f61-9811-fefe227299fb-maas-api-service-ca\") pod \"f65998f0-8e70-4f61-9811-fefe227299fb\" (UID: \"f65998f0-8e70-4f61-9811-fefe227299fb\") " Sep 15 20:56:21.649696 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:21.649675 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65998f0-8e70-4f61-9811-fefe227299fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f65998f0-8e70-4f61-9811-fefe227299fb" (UID: "f65998f0-8e70-4f61-9811-fefe227299fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:56:21.750668 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:21.750636 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65998f0-8e70-4f61-9811-fefe227299fb-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:22.317682 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:22.317653 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" Sep 15 20:56:22.317682 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:22.317673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw" event={"ID":"f65998f0-8e70-4f61-9811-fefe227299fb","Type":"ContainerDied","Data":"91fd132ecd7cb9d983fe333556a8ce5c81cb57374be73d4e09d48e1204de5d65"} Sep 15 20:56:22.317898 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:22.317711 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91fd132ecd7cb9d983fe333556a8ce5c81cb57374be73d4e09d48e1204de5d65" Sep 15 20:56:29.156554 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.156524 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt"] Sep 15 20:56:29.158944 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.156753 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" podUID="64fd895a-ebb4-4a96-b660-a00f4094620a" containerName="payload-processing" containerID="cri-o://91937c2f36c752c63f681af167624d146f6191d03be1ffd4b6b319a9994b92c2" gracePeriod=30 Sep 15 20:56:29.317582 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.317548 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw"] Sep 15 20:56:29.320241 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.320219 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7948686-rnwpw"] Sep 15 20:56:29.368414 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.368338 2579 generic.go:356] "Generic (PLEG): container finished" podID="64fd895a-ebb4-4a96-b660-a00f4094620a" containerID="91937c2f36c752c63f681af167624d146f6191d03be1ffd4b6b319a9994b92c2" exitCode=0 Sep 15 20:56:29.368414 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.368401 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" event={"ID":"64fd895a-ebb4-4a96-b660-a00f4094620a","Type":"ContainerDied","Data":"91937c2f36c752c63f681af167624d146f6191d03be1ffd4b6b319a9994b92c2"} Sep 15 20:56:29.370187 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.370164 2579 generic.go:356] "Generic (PLEG): container finished" podID="180ed213-5e9a-4eab-ac16-2cb3fae30eec" containerID="d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9" exitCode=0 Sep 15 20:56:29.370298 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.370222 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" event={"ID":"180ed213-5e9a-4eab-ac16-2cb3fae30eec","Type":"ContainerDied","Data":"d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9"} Sep 15 20:56:29.370819 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.370802 2579 scope.go:117] "RemoveContainer" containerID="d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9" Sep 15 20:56:29.372042 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.372016 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:56:29.372114 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.372083 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:56:29.372154 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.372113 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:56:29.447418 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.447380 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:29.461097 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.461068 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g"] Sep 15 20:56:29.526253 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.526218 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd7qh\" (UniqueName: \"kubernetes.io/projected/64fd895a-ebb4-4a96-b660-a00f4094620a-kube-api-access-pd7qh\") pod \"64fd895a-ebb4-4a96-b660-a00f4094620a\" (UID: \"64fd895a-ebb4-4a96-b660-a00f4094620a\") " Sep 15 20:56:29.526446 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.526313 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fd895a-ebb4-4a96-b660-a00f4094620a-plugins-config-volume\") pod \"64fd895a-ebb4-4a96-b660-a00f4094620a\" (UID: \"64fd895a-ebb4-4a96-b660-a00f4094620a\") " Sep 15 20:56:29.526758 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.526731 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fd895a-ebb4-4a96-b660-a00f4094620a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64fd895a-ebb4-4a96-b660-a00f4094620a" (UID: "64fd895a-ebb4-4a96-b660-a00f4094620a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:56:29.527151 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.527130 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fd895a-ebb4-4a96-b660-a00f4094620a-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:29.529922 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.529880 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fd895a-ebb4-4a96-b660-a00f4094620a-kube-api-access-pd7qh" (OuterVolumeSpecName: "kube-api-access-pd7qh") pod "64fd895a-ebb4-4a96-b660-a00f4094620a" (UID: "64fd895a-ebb4-4a96-b660-a00f4094620a"). InnerVolumeSpecName "kube-api-access-pd7qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:56:29.627732 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:29.627699 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd7qh\" (UniqueName: \"kubernetes.io/projected/64fd895a-ebb4-4a96-b660-a00f4094620a-kube-api-access-pd7qh\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:30.377885 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:30.377851 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" Sep 15 20:56:30.378335 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:30.377879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt" event={"ID":"64fd895a-ebb4-4a96-b660-a00f4094620a","Type":"ContainerDied","Data":"462d5746ae6e2bac20c8717ade91dd26fdafb234fb9f68d897523b55cf54a14c"} Sep 15 20:56:30.378335 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:30.377950 2579 scope.go:117] "RemoveContainer" containerID="91937c2f36c752c63f681af167624d146f6191d03be1ffd4b6b319a9994b92c2" Sep 15 20:56:30.380226 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:30.380200 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" event={"ID":"180ed213-5e9a-4eab-ac16-2cb3fae30eec","Type":"ContainerStarted","Data":"c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18"} Sep 15 20:56:30.380368 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:30.380318 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" podUID="180ed213-5e9a-4eab-ac16-2cb3fae30eec" containerName="istio-proxy" containerID="cri-o://c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18" gracePeriod=30 Sep 15 20:56:30.419847 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:30.419816 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt"] Sep 15 20:56:30.433575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:30.433547 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7948686-8676c9cdd9-fw5mt"] Sep 15 20:56:31.312437 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:31.312407 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64fd895a-ebb4-4a96-b660-a00f4094620a" path="/var/lib/kubelet/pods/64fd895a-ebb4-4a96-b660-a00f4094620a/volumes" Sep 15 20:56:31.312824 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:31.312808 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f65998f0-8e70-4f61-9811-fefe227299fb" path="/var/lib/kubelet/pods/f65998f0-8e70-4f61-9811-fefe227299fb/volumes" Sep 15 20:56:33.188630 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:33.188600 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:34.138568 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.138524 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" podUID="4a04c99b-a1b2-4c70-9186-034a6c79f81b" containerName="main" containerID="cri-o://fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30" gracePeriod=45 Sep 15 20:56:34.392631 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.392570 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:56:34.405481 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.405451 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a04c99b-a1b2-4c70-9186-034a6c79f81b" containerID="fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30" exitCode=0 Sep 15 20:56:34.405636 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.405571 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" Sep 15 20:56:34.405636 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.405607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" event={"ID":"4a04c99b-a1b2-4c70-9186-034a6c79f81b","Type":"ContainerDied","Data":"fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30"} Sep 15 20:56:34.405764 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.405659 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq" event={"ID":"4a04c99b-a1b2-4c70-9186-034a6c79f81b","Type":"ContainerDied","Data":"0c2d1ef79af28bbd5a67b1dbe372f4b31927194f74c3c275a361a1ef7ecf71b4"} Sep 15 20:56:34.405764 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.405682 2579 scope.go:117] "RemoveContainer" containerID="fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30" Sep 15 20:56:34.416303 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.416281 2579 scope.go:117] "RemoveContainer" containerID="fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30" Sep 15 20:56:34.416671 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:56:34.416648 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30\": container with ID starting with fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30 not found: ID does not exist" containerID="fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30" Sep 15 20:56:34.416788 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.416684 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30"} err="failed to get container status \"fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30\": rpc error: code = NotFound desc = could not find container \"fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30\": container with ID starting with fe6012435f4e328611f956b6f1155eda4f5560df05d2387bd34e30f26aa98e30 not found: ID does not exist" Sep 15 20:56:34.469019 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.468987 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-model-cache\") pod \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " Sep 15 20:56:34.469215 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469037 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lg8v\" (UniqueName: \"kubernetes.io/projected/4a04c99b-a1b2-4c70-9186-034a6c79f81b-kube-api-access-8lg8v\") pod \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " Sep 15 20:56:34.469215 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469068 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tmp-dir\") pod \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " Sep 15 20:56:34.469215 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469105 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-dshm\") pod \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " Sep 15 20:56:34.469215 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469201 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-home\") pod \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " Sep 15 20:56:34.469438 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469263 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a04c99b-a1b2-4c70-9186-034a6c79f81b" (UID: "4a04c99b-a1b2-4c70-9186-034a6c79f81b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:34.469438 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469289 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tls-certs\") pod \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\" (UID: \"4a04c99b-a1b2-4c70-9186-034a6c79f81b\") " Sep 15 20:56:34.469438 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469363 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a04c99b-a1b2-4c70-9186-034a6c79f81b" (UID: "4a04c99b-a1b2-4c70-9186-034a6c79f81b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:34.469612 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469476 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-home" (OuterVolumeSpecName: "home") pod "4a04c99b-a1b2-4c70-9186-034a6c79f81b" (UID: "4a04c99b-a1b2-4c70-9186-034a6c79f81b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:34.469695 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469671 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:34.469760 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469702 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:34.469760 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.469719 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:34.471668 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.471648 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a04c99b-a1b2-4c70-9186-034a6c79f81b" (UID: "4a04c99b-a1b2-4c70-9186-034a6c79f81b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:56:34.471865 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.471849 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-dshm" (OuterVolumeSpecName: "dshm") pod "4a04c99b-a1b2-4c70-9186-034a6c79f81b" (UID: "4a04c99b-a1b2-4c70-9186-034a6c79f81b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:34.471939 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.471887 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a04c99b-a1b2-4c70-9186-034a6c79f81b-kube-api-access-8lg8v" (OuterVolumeSpecName: "kube-api-access-8lg8v") pod "4a04c99b-a1b2-4c70-9186-034a6c79f81b" (UID: "4a04c99b-a1b2-4c70-9186-034a6c79f81b"). InnerVolumeSpecName "kube-api-access-8lg8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:56:34.570767 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.570731 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a04c99b-a1b2-4c70-9186-034a6c79f81b-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:34.570767 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.570761 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lg8v\" (UniqueName: \"kubernetes.io/projected/4a04c99b-a1b2-4c70-9186-034a6c79f81b-kube-api-access-8lg8v\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:34.570767 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.570771 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a04c99b-a1b2-4c70-9186-034a6c79f81b-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:34.726362 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.726332 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq"] Sep 15 20:56:34.729120 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:34.729094 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b9ff68/autores-model-b9ff68-kserve-76bfcdb8f9-cs4bq"] Sep 15 20:56:35.312178 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.312138 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a04c99b-a1b2-4c70-9186-034a6c79f81b" path="/var/lib/kubelet/pods/4a04c99b-a1b2-4c70-9186-034a6c79f81b/volumes" Sep 15 20:56:35.624683 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.624653 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:35.683048 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683017 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhrj4\" (UniqueName: \"kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-kube-api-access-qhrj4\") pod \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " Sep 15 20:56:35.683247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683065 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istiod-ca-cert\") pod \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " Sep 15 20:56:35.683247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683094 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-data\") pod \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " Sep 15 20:56:35.683247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683121 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-envoy\") pod \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " Sep 15 20:56:35.683247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683176 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-podinfo\") pod \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " Sep 15 20:56:35.683247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683205 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-credential-socket\") pod \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " Sep 15 20:56:35.683247 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683242 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-socket\") pod \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " Sep 15 20:56:35.683633 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683288 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-token\") pod \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " Sep 15 20:56:35.683633 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683350 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-certs\") pod \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\" (UID: \"180ed213-5e9a-4eab-ac16-2cb3fae30eec\") " Sep 15 20:56:35.683633 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683360 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-data" (OuterVolumeSpecName: "istio-data") pod "180ed213-5e9a-4eab-ac16-2cb3fae30eec" (UID: "180ed213-5e9a-4eab-ac16-2cb3fae30eec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:35.683633 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683478 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "180ed213-5e9a-4eab-ac16-2cb3fae30eec" (UID: "180ed213-5e9a-4eab-ac16-2cb3fae30eec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:35.683633 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683578 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "180ed213-5e9a-4eab-ac16-2cb3fae30eec" (UID: "180ed213-5e9a-4eab-ac16-2cb3fae30eec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:35.683910 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683729 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "180ed213-5e9a-4eab-ac16-2cb3fae30eec" (UID: "180ed213-5e9a-4eab-ac16-2cb3fae30eec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:35.683910 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683741 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:35.683910 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683759 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:35.683910 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683772 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:35.684281 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.683921 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "180ed213-5e9a-4eab-ac16-2cb3fae30eec" (UID: "180ed213-5e9a-4eab-ac16-2cb3fae30eec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:56:35.685582 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.685551 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "180ed213-5e9a-4eab-ac16-2cb3fae30eec" (UID: "180ed213-5e9a-4eab-ac16-2cb3fae30eec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 20:56:35.685582 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.685560 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "180ed213-5e9a-4eab-ac16-2cb3fae30eec" (UID: "180ed213-5e9a-4eab-ac16-2cb3fae30eec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:56:35.685739 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.685625 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-token" (OuterVolumeSpecName: "istio-token") pod "180ed213-5e9a-4eab-ac16-2cb3fae30eec" (UID: "180ed213-5e9a-4eab-ac16-2cb3fae30eec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:56:35.685739 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.685724 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-kube-api-access-qhrj4" (OuterVolumeSpecName: "kube-api-access-qhrj4") pod "180ed213-5e9a-4eab-ac16-2cb3fae30eec" (UID: "180ed213-5e9a-4eab-ac16-2cb3fae30eec"). InnerVolumeSpecName "kube-api-access-qhrj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:56:35.784367 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.784334 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:35.784367 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.784365 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhrj4\" (UniqueName: \"kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-kube-api-access-qhrj4\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:35.784367 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.784377 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:35.784622 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.784386 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:35.784622 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.784394 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:35.784622 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:35.784402 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/180ed213-5e9a-4eab-ac16-2cb3fae30eec-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:56:36.420092 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.420062 2579 generic.go:356] "Generic (PLEG): container finished" podID="180ed213-5e9a-4eab-ac16-2cb3fae30eec" containerID="c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18" exitCode=0 Sep 15 20:56:36.420271 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.420116 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" Sep 15 20:56:36.420271 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.420122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" event={"ID":"180ed213-5e9a-4eab-ac16-2cb3fae30eec","Type":"ContainerDied","Data":"c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18"} Sep 15 20:56:36.420271 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.420177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g" event={"ID":"180ed213-5e9a-4eab-ac16-2cb3fae30eec","Type":"ContainerDied","Data":"f601963b99778ed19df5ac9e04b8558cf3e99a105d52966fc0e7d8809fb8b1b2"} Sep 15 20:56:36.420271 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.420200 2579 scope.go:117] "RemoveContainer" containerID="c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18" Sep 15 20:56:36.429581 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.429564 2579 scope.go:117] "RemoveContainer" containerID="d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9" Sep 15 20:56:36.437692 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.437672 2579 scope.go:117] "RemoveContainer" containerID="c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18" Sep 15 20:56:36.437950 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:56:36.437931 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18\": container with ID starting with c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18 not found: ID does not exist" containerID="c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18" Sep 15 20:56:36.438000 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.437958 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18"} err="failed to get container status \"c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18\": rpc error: code = NotFound desc = could not find container \"c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18\": container with ID starting with c734ae5f64ea79eb6d06bdc2f19fef031a2b1ccc67a52e24508cca6411c90d18 not found: ID does not exist" Sep 15 20:56:36.438000 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.437973 2579 scope.go:117] "RemoveContainer" containerID="d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9" Sep 15 20:56:36.438223 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:56:36.438208 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9\": container with ID starting with d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9 not found: ID does not exist" containerID="d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9" Sep 15 20:56:36.438274 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.438225 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9"} err="failed to get container status \"d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9\": rpc error: code = NotFound desc = could not find container \"d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9\": container with ID starting with d05be59f34dc4913c415335199dba5b241f9d2c76c34bc1558da632ee4b2abd9 not found: ID does not exist" Sep 15 20:56:36.471351 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.471314 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g"] Sep 15 20:56:36.493235 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:36.493200 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a7948686-openshift-default-6fbc587fcb-pxw2g"] Sep 15 20:56:37.311739 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:56:37.311707 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="180ed213-5e9a-4eab-ac16-2cb3fae30eec" path="/var/lib/kubelet/pods/180ed213-5e9a-4eab-ac16-2cb3fae30eec/volumes" Sep 15 20:57:08.547757 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.547724 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b"] Sep 15 20:57:08.548224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548104 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64fd895a-ebb4-4a96-b660-a00f4094620a" containerName="payload-processing" Sep 15 20:57:08.548224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548115 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="64fd895a-ebb4-4a96-b660-a00f4094620a" containerName="payload-processing" Sep 15 20:57:08.548224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548126 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180ed213-5e9a-4eab-ac16-2cb3fae30eec" containerName="istio-proxy" Sep 15 20:57:08.548224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548131 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="180ed213-5e9a-4eab-ac16-2cb3fae30eec" containerName="istio-proxy" Sep 15 20:57:08.548224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548150 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180ed213-5e9a-4eab-ac16-2cb3fae30eec" containerName="istio-proxy" Sep 15 20:57:08.548224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548155 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="180ed213-5e9a-4eab-ac16-2cb3fae30eec" containerName="istio-proxy" Sep 15 20:57:08.548224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548163 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65998f0-8e70-4f61-9811-fefe227299fb" containerName="revoke-keys" Sep 15 20:57:08.548224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548167 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65998f0-8e70-4f61-9811-fefe227299fb" containerName="revoke-keys" Sep 15 20:57:08.548224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548175 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a04c99b-a1b2-4c70-9186-034a6c79f81b" containerName="main" Sep 15 20:57:08.548224 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548180 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a04c99b-a1b2-4c70-9186-034a6c79f81b" containerName="main" Sep 15 20:57:08.548549 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548231 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="64fd895a-ebb4-4a96-b660-a00f4094620a" containerName="payload-processing" Sep 15 20:57:08.548549 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548239 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a04c99b-a1b2-4c70-9186-034a6c79f81b" containerName="main" Sep 15 20:57:08.548549 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548247 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="180ed213-5e9a-4eab-ac16-2cb3fae30eec" containerName="istio-proxy" Sep 15 20:57:08.548549 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.548254 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65998f0-8e70-4f61-9811-fefe227299fb" containerName="revoke-keys" Sep 15 20:57:08.552759 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.552744 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.558236 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.558215 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-41588d-openshift-default-dockercfg-9dpw4\"" Sep 15 20:57:08.565443 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.565419 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b"] Sep 15 20:57:08.595912 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.595878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdg6\" (UniqueName: \"kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-kube-api-access-ctdg6\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.596065 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.595915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-socket\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.596065 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.595945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-token\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.596065 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.595993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-podinfo\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.596177 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.596074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-certs\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.596177 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.596151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828414c3-79de-4b80-8fc2-fca21628c9a1-istiod-ca-cert\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.596258 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.596181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-credential-socket\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.596258 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.596196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-envoy\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.596258 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.596210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-data\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.697655 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.697621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdg6\" (UniqueName: \"kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-kube-api-access-ctdg6\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.697655 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.697654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-socket\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.697920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.697678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-token\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.697920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.697696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-podinfo\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.697920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.697741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-certs\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.697920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.697794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828414c3-79de-4b80-8fc2-fca21628c9a1-istiod-ca-cert\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.697920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.697824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-credential-socket\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.697920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.697846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-envoy\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.697920 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.697868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-data\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.698259 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.698109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-socket\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.698259 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.698163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-certs\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.698259 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.698195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-credential-socket\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.698362 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.698258 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-data\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.698524 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.698479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828414c3-79de-4b80-8fc2-fca21628c9a1-istiod-ca-cert\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.700258 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.700232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-envoy\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.700394 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.700361 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-podinfo\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.707017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.706994 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-token\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.707146 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.707130 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdg6\" (UniqueName: \"kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-kube-api-access-ctdg6\") pod \"e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:08.862043 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:08.861962 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:09.142146 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:09.142122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b"] Sep 15 20:57:09.144167 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:57:09.144139 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828414c3_79de_4b80_8fc2_fca21628c9a1.slice/crio-33b8c5075b954dcbfd677f0bf09eab371db8563a20d9a9fdc6764bb37de6a175 WatchSource:0}: Error finding container 33b8c5075b954dcbfd677f0bf09eab371db8563a20d9a9fdc6764bb37de6a175: Status 404 returned error can't find the container with id 33b8c5075b954dcbfd677f0bf09eab371db8563a20d9a9fdc6764bb37de6a175 Sep 15 20:57:09.146410 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:09.146378 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:57:09.146479 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:09.146452 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:57:09.146541 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:09.146482 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:57:09.585743 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:09.585703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" event={"ID":"828414c3-79de-4b80-8fc2-fca21628c9a1","Type":"ContainerStarted","Data":"dc6100193aa2eb4a707d14e1850b647d388f219256d1e958acf4e3d3c46826f9"} Sep 15 20:57:09.586139 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:09.585757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" event={"ID":"828414c3-79de-4b80-8fc2-fca21628c9a1","Type":"ContainerStarted","Data":"33b8c5075b954dcbfd677f0bf09eab371db8563a20d9a9fdc6764bb37de6a175"} Sep 15 20:57:09.613389 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:09.613332 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" podStartSLOduration=1.6133123839999999 podStartE2EDuration="1.613312384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:57:09.611847028 +0000 UTC m=+1226.805800495" watchObservedRunningTime="2026-09-15 20:57:09.613312384 +0000 UTC m=+1226.807265839" Sep 15 20:57:09.862385 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:09.862287 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:10.867901 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:10.867876 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:11.596610 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:11.596581 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:11.597702 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:11.597684 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:16.440574 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.440541 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4"] Sep 15 20:57:16.441176 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.441156 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="180ed213-5e9a-4eab-ac16-2cb3fae30eec" containerName="istio-proxy" Sep 15 20:57:16.444299 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.444278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:57:16.446623 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.446601 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-41588d\"" Sep 15 20:57:16.446747 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.446607 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-41588d-dockercfg-57rk4\"" Sep 15 20:57:16.455107 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.455082 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4"] Sep 15 20:57:16.570088 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.570053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pdhc\" (UniqueName: \"kubernetes.io/projected/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-kube-api-access-8pdhc\") pod \"payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4\" (UID: \"1a9ff674-d9c5-413b-9a08-79c5869aa7f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:57:16.570290 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.570180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4\" (UID: \"1a9ff674-d9c5-413b-9a08-79c5869aa7f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:57:16.671084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.671048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4\" (UID: \"1a9ff674-d9c5-413b-9a08-79c5869aa7f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:57:16.671269 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.671162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pdhc\" (UniqueName: \"kubernetes.io/projected/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-kube-api-access-8pdhc\") pod \"payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4\" (UID: \"1a9ff674-d9c5-413b-9a08-79c5869aa7f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:57:16.671832 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.671808 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4\" (UID: \"1a9ff674-d9c5-413b-9a08-79c5869aa7f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:57:16.679214 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.679190 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pdhc\" (UniqueName: \"kubernetes.io/projected/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-kube-api-access-8pdhc\") pod \"payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4\" (UID: \"1a9ff674-d9c5-413b-9a08-79c5869aa7f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:57:16.758171 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:16.758085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:57:17.028950 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:17.028859 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4"] Sep 15 20:57:17.031274 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:57:17.031252 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9ff674_d9c5_413b_9a08_79c5869aa7f1.slice/crio-ebc5308ec278db00d0ba8bd3816a8230dc8a2f6bdc889c3ed2f0213bd20a454e WatchSource:0}: Error finding container ebc5308ec278db00d0ba8bd3816a8230dc8a2f6bdc889c3ed2f0213bd20a454e: Status 404 returned error can't find the container with id ebc5308ec278db00d0ba8bd3816a8230dc8a2f6bdc889c3ed2f0213bd20a454e Sep 15 20:57:17.626999 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:17.626955 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" event={"ID":"1a9ff674-d9c5-413b-9a08-79c5869aa7f1","Type":"ContainerStarted","Data":"24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6"} Sep 15 20:57:17.627484 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:17.627009 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" event={"ID":"1a9ff674-d9c5-413b-9a08-79c5869aa7f1","Type":"ContainerStarted","Data":"ebc5308ec278db00d0ba8bd3816a8230dc8a2f6bdc889c3ed2f0213bd20a454e"} Sep 15 20:57:17.627484 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:17.627150 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:57:17.644759 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:17.644703 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" podStartSLOduration=1.644686329 podStartE2EDuration="1.644686329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:57:17.643173857 +0000 UTC m=+1234.837127313" watchObservedRunningTime="2026-09-15 20:57:17.644686329 +0000 UTC m=+1234.838639784" Sep 15 20:57:23.676620 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.676588 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n"] Sep 15 20:57:23.680226 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.680208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.684072 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.684057 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"premium-w4-16bde4-kserve-self-signed-certs\"" Sep 15 20:57:23.696933 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.696906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n"] Sep 15 20:57:23.735450 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.735419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b174a8a-fe9f-4360-b931-0c9016c8832c-tls-certs\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.735641 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.735455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9g7\" (UniqueName: \"kubernetes.io/projected/3b174a8a-fe9f-4360-b931-0c9016c8832c-kube-api-access-hd9g7\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.735641 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.735487 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-model-cache\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.735777 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.735641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-dshm\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.735777 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.735686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-tmp-dir\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.735777 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.735732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-home\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.836802 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.836768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-dshm\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.836975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.836808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-tmp-dir\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.836975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.836833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-home\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.836975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.836900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b174a8a-fe9f-4360-b931-0c9016c8832c-tls-certs\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.836975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.836928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9g7\" (UniqueName: \"kubernetes.io/projected/3b174a8a-fe9f-4360-b931-0c9016c8832c-kube-api-access-hd9g7\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.836975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.836971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-model-cache\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.837323 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.837281 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-tmp-dir\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.837323 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.837311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-home\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.837471 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.837341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-model-cache\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.839217 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.839191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-dshm\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.839519 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.839473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b174a8a-fe9f-4360-b931-0c9016c8832c-tls-certs\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.845820 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.845799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9g7\" (UniqueName: \"kubernetes.io/projected/3b174a8a-fe9f-4360-b931-0c9016c8832c-kube-api-access-hd9g7\") pod \"premium-w4-16bde4-kserve-5dc46c8976-sw46n\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:23.989400 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:23.989316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:24.278390 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:24.278362 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n"] Sep 15 20:57:24.280983 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:57:24.280953 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b174a8a_fe9f_4360_b931_0c9016c8832c.slice/crio-fbdf4d988d5b3a6227e7ee9f9d268b4231263dc2865a87069ac449edc3d1fbcd WatchSource:0}: Error finding container fbdf4d988d5b3a6227e7ee9f9d268b4231263dc2865a87069ac449edc3d1fbcd: Status 404 returned error can't find the container with id fbdf4d988d5b3a6227e7ee9f9d268b4231263dc2865a87069ac449edc3d1fbcd Sep 15 20:57:24.665240 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:24.665135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" event={"ID":"3b174a8a-fe9f-4360-b931-0c9016c8832c","Type":"ContainerStarted","Data":"552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea"} Sep 15 20:57:24.665240 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:24.665174 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" event={"ID":"3b174a8a-fe9f-4360-b931-0c9016c8832c","Type":"ContainerStarted","Data":"fbdf4d988d5b3a6227e7ee9f9d268b4231263dc2865a87069ac449edc3d1fbcd"} Sep 15 20:57:28.634693 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:28.634659 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:57:28.668597 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:28.668547 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" podStartSLOduration=5.668531463 podStartE2EDuration="5.668531463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:57:24.707258975 +0000 UTC m=+1241.901212431" watchObservedRunningTime="2026-09-15 20:57:28.668531463 +0000 UTC m=+1245.862484953" Sep 15 20:57:33.990084 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:33.990044 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:33.990478 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:33.990099 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:34.003633 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:34.003605 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:34.741600 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:34.741517 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 20:57:36.741615 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:36.741584 2579 generic.go:356] "Generic (PLEG): container finished" podID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerID="dc6100193aa2eb4a707d14e1850b647d388f219256d1e958acf4e3d3c46826f9" exitCode=0 Sep 15 20:57:36.742070 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:36.741634 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" event={"ID":"828414c3-79de-4b80-8fc2-fca21628c9a1","Type":"ContainerDied","Data":"dc6100193aa2eb4a707d14e1850b647d388f219256d1e958acf4e3d3c46826f9"} Sep 15 20:57:36.742070 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:36.741988 2579 scope.go:117] "RemoveContainer" containerID="dc6100193aa2eb4a707d14e1850b647d388f219256d1e958acf4e3d3c46826f9" Sep 15 20:57:36.743128 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:36.743099 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:57:36.743200 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:36.743168 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:57:36.743237 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:36.743196 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:57:37.748968 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:37.748934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" event={"ID":"828414c3-79de-4b80-8fc2-fca21628c9a1","Type":"ContainerStarted","Data":"d25ed7a50c27601a5105cba3853f8f4e5a99561cc288456c7d311bbf8bde03ae"} Sep 15 20:57:37.862883 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:37.862845 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:37.864017 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:37.863990 2579 patch_prober.go:28] interesting pod/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 15 20:57:37.864129 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:37.864052 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 15 20:57:38.863130 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:38.863097 2579 patch_prober.go:28] interesting pod/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 15 20:57:38.863581 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:38.863150 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 15 20:57:39.862816 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:39.862783 2579 patch_prober.go:28] interesting pod/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 15 20:57:39.862993 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:39.862835 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 15 20:57:40.862482 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:40.862444 2579 patch_prober.go:28] interesting pod/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 15 20:57:40.862855 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:40.862527 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 15 20:57:41.597372 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:41.597331 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:41.862715 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:41.862625 2579 patch_prober.go:28] interesting pod/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 15 20:57:41.862715 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:41.862684 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 15 20:57:42.886913 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:42.886880 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:42.889278 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:42.889255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:57:58.962056 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:58.962023 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5"] Sep 15 20:57:58.965766 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:58.965747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:58.970714 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:58.970688 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-988a09\"/\"test-model-988a09-kserve-self-signed-certs\"" Sep 15 20:57:58.970714 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:58.970710 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-988a09\"/\"openshift-service-ca.crt\"" Sep 15 20:57:58.970897 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:58.970730 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-988a09\"/\"kube-root-ca.crt\"" Sep 15 20:57:58.971041 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:58.971021 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-988a09\"/\"default-dockercfg-thwsc\"" Sep 15 20:57:58.983901 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:58.983876 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5"] Sep 15 20:57:59.097477 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.097446 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-model-cache\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.097477 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.097480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tmp-dir\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.097711 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.097578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-dshm\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.097711 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.097625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-home\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.097711 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.097642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tls-certs\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.097711 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.097695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kg54\" (UniqueName: \"kubernetes.io/projected/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-kube-api-access-7kg54\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.198838 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.198802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kg54\" (UniqueName: \"kubernetes.io/projected/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-kube-api-access-7kg54\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.199042 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.198870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-model-cache\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.199042 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.198897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tmp-dir\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.199042 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.198954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-dshm\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.199042 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.198993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-home\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.199042 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.199012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tls-certs\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.199312 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.199289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-model-cache\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.199380 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.199360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tmp-dir\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.199439 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.199379 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-home\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.201314 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.201294 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-dshm\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.201591 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.201574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tls-certs\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.207784 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.207760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kg54\" (UniqueName: \"kubernetes.io/projected/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-kube-api-access-7kg54\") pod \"test-model-988a09-kserve-86bf4dffbc-pgzr5\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.274872 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.274846 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:57:59.550244 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.550220 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5"] Sep 15 20:57:59.552142 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:57:59.552115 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e0e80e5_72ec_4c12_a2bc_1f9e01d8803a.slice/crio-d5ce1197127b1cb895c6ad31484e6dede1e229d5f997e37e6dde4aad843e1418 WatchSource:0}: Error finding container d5ce1197127b1cb895c6ad31484e6dede1e229d5f997e37e6dde4aad843e1418: Status 404 returned error can't find the container with id d5ce1197127b1cb895c6ad31484e6dede1e229d5f997e37e6dde4aad843e1418 Sep 15 20:57:59.865324 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.865218 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" event={"ID":"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a","Type":"ContainerStarted","Data":"eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4"} Sep 15 20:57:59.865324 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.865280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" event={"ID":"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a","Type":"ContainerStarted","Data":"d5ce1197127b1cb895c6ad31484e6dede1e229d5f997e37e6dde4aad843e1418"} Sep 15 20:57:59.882266 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:57:59.882217 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" podStartSLOduration=1.882202542 podStartE2EDuration="1.882202542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:57:59.881792496 +0000 UTC m=+1277.075745951" watchObservedRunningTime="2026-09-15 20:57:59.882202542 +0000 UTC m=+1277.076155996" Sep 15 20:58:09.275589 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:09.275546 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:58:09.276030 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:09.275613 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:58:09.288291 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:09.288271 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:58:09.928128 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:09.928101 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 20:58:19.512483 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.512450 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf"] Sep 15 20:58:19.515575 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.515553 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:58:19.518016 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.518000 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c103b8e8-dockercfg-k8x59\"" Sep 15 20:58:19.518095 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.518075 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c103b8e8\"" Sep 15 20:58:19.526596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.526576 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf"] Sep 15 20:58:19.597464 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.597429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf\" (UID: \"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:58:19.597664 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.597472 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjq9t\" (UniqueName: \"kubernetes.io/projected/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-kube-api-access-fjq9t\") pod \"payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf\" (UID: \"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:58:19.698750 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.698716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf\" (UID: \"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:58:19.698915 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.698760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjq9t\" (UniqueName: \"kubernetes.io/projected/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-kube-api-access-fjq9t\") pod \"payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf\" (UID: \"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:58:19.699348 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.699331 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf\" (UID: \"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:58:19.706696 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.706666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjq9t\" (UniqueName: \"kubernetes.io/projected/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-kube-api-access-fjq9t\") pod \"payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf\" (UID: \"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:58:19.827239 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:19.827167 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:58:20.098419 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:20.098393 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf"] Sep 15 20:58:20.100642 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:58:20.100615 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2b3d02_1e1b_4d32_bbb3_f6a40f4e2f0e.slice/crio-4b542be854fc9bd013bffe8724d57039e8810f38e1a13841bffa0ec334c2537f WatchSource:0}: Error finding container 4b542be854fc9bd013bffe8724d57039e8810f38e1a13841bffa0ec334c2537f: Status 404 returned error can't find the container with id 4b542be854fc9bd013bffe8724d57039e8810f38e1a13841bffa0ec334c2537f Sep 15 20:58:20.977479 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:20.977434 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" event={"ID":"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e","Type":"ContainerStarted","Data":"34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9"} Sep 15 20:58:20.977919 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:20.977523 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:58:20.977919 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:20.977550 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" event={"ID":"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e","Type":"ContainerStarted","Data":"4b542be854fc9bd013bffe8724d57039e8810f38e1a13841bffa0ec334c2537f"} Sep 15 20:58:20.995473 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:20.995426 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" podStartSLOduration=1.995406432 podStartE2EDuration="1.995406432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:20.994563472 +0000 UTC m=+1298.188516926" watchObservedRunningTime="2026-09-15 20:58:20.995406432 +0000 UTC m=+1298.189359891" Sep 15 20:58:31.986198 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:31.986169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:58:52.867918 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:52.867884 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6"] Sep 15 20:58:52.871636 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:52.871619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" Sep 15 20:58:52.873652 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:52.873628 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 20:58:52.878453 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:52.878428 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6"] Sep 15 20:58:53.011105 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:53.011052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca6804c-6454-4710-8a0c-69e245d157d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6\" (UID: \"fca6804c-6454-4710-8a0c-69e245d157d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" Sep 15 20:58:53.112751 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:53.112714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca6804c-6454-4710-8a0c-69e245d157d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6\" (UID: \"fca6804c-6454-4710-8a0c-69e245d157d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" Sep 15 20:58:53.113352 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:53.113333 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca6804c-6454-4710-8a0c-69e245d157d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6\" (UID: \"fca6804c-6454-4710-8a0c-69e245d157d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" Sep 15 20:58:53.182560 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:53.182479 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" Sep 15 20:58:53.454144 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:53.454120 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6"] Sep 15 20:58:53.456863 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:58:53.456834 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca6804c_6454_4710_8a0c_69e245d157d9.slice/crio-199691051beee48194d3d30515c78692a18a54643aeeb7160a70dc56db56148e WatchSource:0}: Error finding container 199691051beee48194d3d30515c78692a18a54643aeeb7160a70dc56db56148e: Status 404 returned error can't find the container with id 199691051beee48194d3d30515c78692a18a54643aeeb7160a70dc56db56148e Sep 15 20:58:54.098663 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.098588 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr"] Sep 15 20:58:54.102134 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.102118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.104242 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.104215 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-41588d\"/\"openshift-service-ca.crt\"" Sep 15 20:58:54.104384 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.104301 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-41588d\"/\"kube-root-ca.crt\"" Sep 15 20:58:54.105204 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.105166 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-41588d\"/\"default-dockercfg-2c9pz\"" Sep 15 20:58:54.105319 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.105246 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-41588d\"/\"explicit-model-41588d-kserve-self-signed-certs\"" Sep 15 20:58:54.112729 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.112709 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr"] Sep 15 20:58:54.157372 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.157343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" event={"ID":"fca6804c-6454-4710-8a0c-69e245d157d9","Type":"ContainerStarted","Data":"5b2c573e316dfbaa7db75b7c1d74112cf3e6a2f823ed1900bdfeb82e3241ca93"} Sep 15 20:58:54.157547 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.157389 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" event={"ID":"fca6804c-6454-4710-8a0c-69e245d157d9","Type":"ContainerStarted","Data":"199691051beee48194d3d30515c78692a18a54643aeeb7160a70dc56db56148e"} Sep 15 20:58:54.176222 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.176170 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" podStartSLOduration=2.176151324 podStartE2EDuration="2.176151324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:54.173790023 +0000 UTC m=+1331.367743503" watchObservedRunningTime="2026-09-15 20:58:54.176151324 +0000 UTC m=+1331.370104782" Sep 15 20:58:54.222454 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.222423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-home\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.222594 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.222461 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwq6\" (UniqueName: \"kubernetes.io/projected/5d939f11-8c94-487b-b67a-875da7b7be21-kube-api-access-tbwq6\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.222594 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.222557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-tmp-dir\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.222681 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.222601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d939f11-8c94-487b-b67a-875da7b7be21-tls-certs\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.222681 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.222629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-dshm\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.222757 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.222704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-model-cache\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.323870 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.323835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-home\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.324052 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.323875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwq6\" (UniqueName: \"kubernetes.io/projected/5d939f11-8c94-487b-b67a-875da7b7be21-kube-api-access-tbwq6\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.324052 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.323930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-tmp-dir\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.324052 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.323953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d939f11-8c94-487b-b67a-875da7b7be21-tls-certs\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.324052 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.323971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-dshm\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.324052 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.324028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-model-cache\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.324355 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.324279 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-home\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.324355 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.324334 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-tmp-dir\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.324449 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.324409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-model-cache\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.326375 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.326352 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-dshm\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.326683 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.326666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d939f11-8c94-487b-b67a-875da7b7be21-tls-certs\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.332323 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.332301 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwq6\" (UniqueName: \"kubernetes.io/projected/5d939f11-8c94-487b-b67a-875da7b7be21-kube-api-access-tbwq6\") pod \"explicit-model-41588d-kserve-6597b6f449-4bdsr\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.412532 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.412431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:58:54.686289 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:54.686264 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr"] Sep 15 20:58:54.688353 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:58:54.688329 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d939f11_8c94_487b_b67a_875da7b7be21.slice/crio-20a32fc5ecdcf8967d1811a57e1e27e5676f29679c175bae9b487b707c07c748 WatchSource:0}: Error finding container 20a32fc5ecdcf8967d1811a57e1e27e5676f29679c175bae9b487b707c07c748: Status 404 returned error can't find the container with id 20a32fc5ecdcf8967d1811a57e1e27e5676f29679c175bae9b487b707c07c748 Sep 15 20:58:55.164937 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:55.164900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" event={"ID":"5d939f11-8c94-487b-b67a-875da7b7be21","Type":"ContainerStarted","Data":"e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578"} Sep 15 20:58:55.165402 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:55.164949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" event={"ID":"5d939f11-8c94-487b-b67a-875da7b7be21","Type":"ContainerStarted","Data":"20a32fc5ecdcf8967d1811a57e1e27e5676f29679c175bae9b487b707c07c748"} Sep 15 20:58:55.166560 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:55.166534 2579 generic.go:356] "Generic (PLEG): container finished" podID="fca6804c-6454-4710-8a0c-69e245d157d9" containerID="5b2c573e316dfbaa7db75b7c1d74112cf3e6a2f823ed1900bdfeb82e3241ca93" exitCode=0 Sep 15 20:58:55.166698 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:55.166599 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" event={"ID":"fca6804c-6454-4710-8a0c-69e245d157d9","Type":"ContainerDied","Data":"5b2c573e316dfbaa7db75b7c1d74112cf3e6a2f823ed1900bdfeb82e3241ca93"} Sep 15 20:58:55.180685 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:55.180641 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" podStartSLOduration=1.180626403 podStartE2EDuration="1.180626403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:55.178938948 +0000 UTC m=+1332.372892426" watchObservedRunningTime="2026-09-15 20:58:55.180626403 +0000 UTC m=+1332.374579893" Sep 15 20:58:56.308369 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:56.308346 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" Sep 15 20:58:56.446275 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:56.446191 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca6804c-6454-4710-8a0c-69e245d157d9-maas-api-service-ca\") pod \"fca6804c-6454-4710-8a0c-69e245d157d9\" (UID: \"fca6804c-6454-4710-8a0c-69e245d157d9\") " Sep 15 20:58:56.446531 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:56.446509 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca6804c-6454-4710-8a0c-69e245d157d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fca6804c-6454-4710-8a0c-69e245d157d9" (UID: "fca6804c-6454-4710-8a0c-69e245d157d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:58:56.446616 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:56.446604 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca6804c-6454-4710-8a0c-69e245d157d9-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:58:57.178473 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:57.178446 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" Sep 15 20:58:57.178659 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:57.178444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6" event={"ID":"fca6804c-6454-4710-8a0c-69e245d157d9","Type":"ContainerDied","Data":"199691051beee48194d3d30515c78692a18a54643aeeb7160a70dc56db56148e"} Sep 15 20:58:57.178659 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:58:57.178553 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="199691051beee48194d3d30515c78692a18a54643aeeb7160a70dc56db56148e" Sep 15 20:59:03.409412 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.409380 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf"] Sep 15 20:59:03.409997 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.409737 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" podUID="fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e" containerName="payload-pre-processing" containerID="cri-o://34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9" gracePeriod=30 Sep 15 20:59:03.567863 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.567831 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6"] Sep 15 20:59:03.570508 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.570477 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c103b8e8-jsxs6"] Sep 15 20:59:03.664688 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.664619 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:59:03.719816 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.719785 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-plugins-config-volume\") pod \"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e\" (UID: \"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e\") " Sep 15 20:59:03.719973 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.719885 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjq9t\" (UniqueName: \"kubernetes.io/projected/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-kube-api-access-fjq9t\") pod \"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e\" (UID: \"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e\") " Sep 15 20:59:03.720219 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.720183 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e" (UID: "fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:59:03.722073 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.722051 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-kube-api-access-fjq9t" (OuterVolumeSpecName: "kube-api-access-fjq9t") pod "fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e" (UID: "fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e"). InnerVolumeSpecName "kube-api-access-fjq9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:59:03.820729 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.820697 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:03.820729 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:03.820724 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjq9t\" (UniqueName: \"kubernetes.io/projected/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e-kube-api-access-fjq9t\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:04.216984 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.216947 2579 generic.go:356] "Generic (PLEG): container finished" podID="fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e" containerID="34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9" exitCode=0 Sep 15 20:59:04.217171 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.216992 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" Sep 15 20:59:04.217171 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.217033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" event={"ID":"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e","Type":"ContainerDied","Data":"34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9"} Sep 15 20:59:04.217171 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.217082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf" event={"ID":"fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e","Type":"ContainerDied","Data":"4b542be854fc9bd013bffe8724d57039e8810f38e1a13841bffa0ec334c2537f"} Sep 15 20:59:04.217171 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.217105 2579 scope.go:117] "RemoveContainer" containerID="34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9" Sep 15 20:59:04.226873 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.226854 2579 scope.go:117] "RemoveContainer" containerID="34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9" Sep 15 20:59:04.227155 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:59:04.227135 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9\": container with ID starting with 34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9 not found: ID does not exist" containerID="34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9" Sep 15 20:59:04.227201 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.227165 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9"} err="failed to get container status \"34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9\": rpc error: code = NotFound desc = could not find container \"34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9\": container with ID starting with 34d8ce2da77f5c5ac599d61b312fdf98721aaf0e472d1d702d700f12ec0db7d9 not found: ID does not exist" Sep 15 20:59:04.239366 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.239341 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf"] Sep 15 20:59:04.244249 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.244228 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c103b8e8-6ff5f54546-4nwqf"] Sep 15 20:59:04.413350 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.413261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:59:04.413350 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.413338 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:59:04.426409 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:04.426380 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:59:05.235091 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:05.235061 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:59:05.312733 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:05.312704 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fca6804c-6454-4710-8a0c-69e245d157d9" path="/var/lib/kubelet/pods/fca6804c-6454-4710-8a0c-69e245d157d9/volumes" Sep 15 20:59:05.313063 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:05.313050 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e" path="/var/lib/kubelet/pods/fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e/volumes" Sep 15 20:59:18.671312 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.671279 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr"] Sep 15 20:59:18.794186 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.794154 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp"] Sep 15 20:59:18.794573 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.794560 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fca6804c-6454-4710-8a0c-69e245d157d9" containerName="revoke-keys" Sep 15 20:59:18.794632 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.794575 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fca6804c-6454-4710-8a0c-69e245d157d9" containerName="revoke-keys" Sep 15 20:59:18.794632 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.794590 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e" containerName="payload-pre-processing" Sep 15 20:59:18.794632 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.794595 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e" containerName="payload-pre-processing" Sep 15 20:59:18.794736 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.794652 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe2b3d02-1e1b-4d32-bbb3-f6a40f4e2f0e" containerName="payload-pre-processing" Sep 15 20:59:18.794736 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.794659 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fca6804c-6454-4710-8a0c-69e245d157d9" containerName="revoke-keys" Sep 15 20:59:18.797405 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.797388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp" Sep 15 20:59:18.799428 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.799404 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 20:59:18.803200 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.803175 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp"] Sep 15 20:59:18.868898 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.868858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019a8674-9df0-4c04-9eb7-b7550244719d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-41588d-c8xjp\" (UID: \"019a8674-9df0-4c04-9eb7-b7550244719d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp" Sep 15 20:59:18.970362 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.970278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019a8674-9df0-4c04-9eb7-b7550244719d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-41588d-c8xjp\" (UID: \"019a8674-9df0-4c04-9eb7-b7550244719d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp" Sep 15 20:59:18.970898 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:18.970878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019a8674-9df0-4c04-9eb7-b7550244719d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-41588d-c8xjp\" (UID: \"019a8674-9df0-4c04-9eb7-b7550244719d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp" Sep 15 20:59:19.111297 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:19.111249 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp" Sep 15 20:59:19.379674 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:19.379651 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp"] Sep 15 20:59:19.381720 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:59:19.381694 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019a8674_9df0_4c04_9eb7_b7550244719d.slice/crio-62630ca9c1bd265cd9f9dc694f740984e4e0106965b6938c5aab80522638999e WatchSource:0}: Error finding container 62630ca9c1bd265cd9f9dc694f740984e4e0106965b6938c5aab80522638999e: Status 404 returned error can't find the container with id 62630ca9c1bd265cd9f9dc694f740984e4e0106965b6938c5aab80522638999e Sep 15 20:59:20.307792 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:20.307754 2579 generic.go:356] "Generic (PLEG): container finished" podID="019a8674-9df0-4c04-9eb7-b7550244719d" containerID="3dabdd57b7e4e53bdb348ff4936c9c6ac2c2b82f6fd120231ca94733dd890954" exitCode=0 Sep 15 20:59:20.308229 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:20.307836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp" event={"ID":"019a8674-9df0-4c04-9eb7-b7550244719d","Type":"ContainerDied","Data":"3dabdd57b7e4e53bdb348ff4936c9c6ac2c2b82f6fd120231ca94733dd890954"} Sep 15 20:59:20.308229 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:20.307879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp" event={"ID":"019a8674-9df0-4c04-9eb7-b7550244719d","Type":"ContainerStarted","Data":"62630ca9c1bd265cd9f9dc694f740984e4e0106965b6938c5aab80522638999e"} Sep 15 20:59:21.447972 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:21.447950 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp" Sep 15 20:59:21.495349 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:21.495320 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019a8674-9df0-4c04-9eb7-b7550244719d-maas-api-service-ca\") pod \"019a8674-9df0-4c04-9eb7-b7550244719d\" (UID: \"019a8674-9df0-4c04-9eb7-b7550244719d\") " Sep 15 20:59:21.495694 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:21.495671 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019a8674-9df0-4c04-9eb7-b7550244719d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "019a8674-9df0-4c04-9eb7-b7550244719d" (UID: "019a8674-9df0-4c04-9eb7-b7550244719d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:59:21.596416 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:21.596332 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019a8674-9df0-4c04-9eb7-b7550244719d-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:22.319614 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:22.319582 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp" Sep 15 20:59:22.319798 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:22.319623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp" event={"ID":"019a8674-9df0-4c04-9eb7-b7550244719d","Type":"ContainerDied","Data":"62630ca9c1bd265cd9f9dc694f740984e4e0106965b6938c5aab80522638999e"} Sep 15 20:59:22.319798 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:22.319670 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62630ca9c1bd265cd9f9dc694f740984e4e0106965b6938c5aab80522638999e" Sep 15 20:59:29.741396 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:29.741359 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4"] Sep 15 20:59:29.741929 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:29.741746 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" podUID="1a9ff674-d9c5-413b-9a08-79c5869aa7f1" containerName="payload-pre-processing" containerID="cri-o://24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6" gracePeriod=30 Sep 15 20:59:29.893004 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:29.892965 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp"] Sep 15 20:59:29.895233 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:29.895195 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41588d-c8xjp"] Sep 15 20:59:30.001136 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.001056 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:59:30.082096 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.082067 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pdhc\" (UniqueName: \"kubernetes.io/projected/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-kube-api-access-8pdhc\") pod \"1a9ff674-d9c5-413b-9a08-79c5869aa7f1\" (UID: \"1a9ff674-d9c5-413b-9a08-79c5869aa7f1\") " Sep 15 20:59:30.082258 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.082124 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-plugins-config-volume\") pod \"1a9ff674-d9c5-413b-9a08-79c5869aa7f1\" (UID: \"1a9ff674-d9c5-413b-9a08-79c5869aa7f1\") " Sep 15 20:59:30.082542 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.082489 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a9ff674-d9c5-413b-9a08-79c5869aa7f1" (UID: "1a9ff674-d9c5-413b-9a08-79c5869aa7f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:59:30.084335 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.084315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-kube-api-access-8pdhc" (OuterVolumeSpecName: "kube-api-access-8pdhc") pod "1a9ff674-d9c5-413b-9a08-79c5869aa7f1" (UID: "1a9ff674-d9c5-413b-9a08-79c5869aa7f1"). InnerVolumeSpecName "kube-api-access-8pdhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:59:30.183758 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.183723 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pdhc\" (UniqueName: \"kubernetes.io/projected/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-kube-api-access-8pdhc\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:30.183758 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.183751 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ff674-d9c5-413b-9a08-79c5869aa7f1-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:30.363953 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.363919 2579 generic.go:356] "Generic (PLEG): container finished" podID="1a9ff674-d9c5-413b-9a08-79c5869aa7f1" containerID="24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6" exitCode=0 Sep 15 20:59:30.364100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.363972 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" Sep 15 20:59:30.364100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.363990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" event={"ID":"1a9ff674-d9c5-413b-9a08-79c5869aa7f1","Type":"ContainerDied","Data":"24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6"} Sep 15 20:59:30.364100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.364043 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4" event={"ID":"1a9ff674-d9c5-413b-9a08-79c5869aa7f1","Type":"ContainerDied","Data":"ebc5308ec278db00d0ba8bd3816a8230dc8a2f6bdc889c3ed2f0213bd20a454e"} Sep 15 20:59:30.364100 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.364064 2579 scope.go:117] "RemoveContainer" containerID="24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6" Sep 15 20:59:30.373109 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.373093 2579 scope.go:117] "RemoveContainer" containerID="24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6" Sep 15 20:59:30.373375 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:59:30.373359 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6\": container with ID starting with 24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6 not found: ID does not exist" containerID="24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6" Sep 15 20:59:30.373421 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.373382 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6"} err="failed to get container status \"24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6\": rpc error: code = NotFound desc = could not find container \"24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6\": container with ID starting with 24733e3bcd25a99798b3c4071d36d57d4e6e0690f33359e63d2382a3fc89a9a6 not found: ID does not exist" Sep 15 20:59:30.387582 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.387559 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4"] Sep 15 20:59:30.391079 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:30.391058 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41588d-844bff4666-jnzx4"] Sep 15 20:59:31.311435 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:31.311400 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="019a8674-9df0-4c04-9eb7-b7550244719d" path="/var/lib/kubelet/pods/019a8674-9df0-4c04-9eb7-b7550244719d/volumes" Sep 15 20:59:31.311835 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:31.311804 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a9ff674-d9c5-413b-9a08-79c5869aa7f1" path="/var/lib/kubelet/pods/1a9ff674-d9c5-413b-9a08-79c5869aa7f1/volumes" Sep 15 20:59:33.693170 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:33.693135 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" podUID="5d939f11-8c94-487b-b67a-875da7b7be21" containerName="main" containerID="cri-o://e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578" gracePeriod=45 Sep 15 20:59:33.936978 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:33.936955 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:59:34.017750 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.017728 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbwq6\" (UniqueName: \"kubernetes.io/projected/5d939f11-8c94-487b-b67a-875da7b7be21-kube-api-access-tbwq6\") pod \"5d939f11-8c94-487b-b67a-875da7b7be21\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " Sep 15 20:59:34.017925 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.017760 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-model-cache\") pod \"5d939f11-8c94-487b-b67a-875da7b7be21\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " Sep 15 20:59:34.017925 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.017793 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-home\") pod \"5d939f11-8c94-487b-b67a-875da7b7be21\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " Sep 15 20:59:34.017925 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.017821 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-tmp-dir\") pod \"5d939f11-8c94-487b-b67a-875da7b7be21\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " Sep 15 20:59:34.017925 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.017878 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d939f11-8c94-487b-b67a-875da7b7be21-tls-certs\") pod \"5d939f11-8c94-487b-b67a-875da7b7be21\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " Sep 15 20:59:34.017925 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.017913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-dshm\") pod \"5d939f11-8c94-487b-b67a-875da7b7be21\" (UID: \"5d939f11-8c94-487b-b67a-875da7b7be21\") " Sep 15 20:59:34.018176 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.018003 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d939f11-8c94-487b-b67a-875da7b7be21" (UID: "5d939f11-8c94-487b-b67a-875da7b7be21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:34.018176 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.018102 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-home" (OuterVolumeSpecName: "home") pod "5d939f11-8c94-487b-b67a-875da7b7be21" (UID: "5d939f11-8c94-487b-b67a-875da7b7be21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:34.018176 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.018114 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d939f11-8c94-487b-b67a-875da7b7be21" (UID: "5d939f11-8c94-487b-b67a-875da7b7be21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:34.018310 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.018293 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:34.018370 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.018312 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:34.018370 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.018321 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:34.020128 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.020102 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-dshm" (OuterVolumeSpecName: "dshm") pod "5d939f11-8c94-487b-b67a-875da7b7be21" (UID: "5d939f11-8c94-487b-b67a-875da7b7be21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:34.020128 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.020114 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d939f11-8c94-487b-b67a-875da7b7be21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d939f11-8c94-487b-b67a-875da7b7be21" (UID: "5d939f11-8c94-487b-b67a-875da7b7be21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:59:34.020265 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.020228 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d939f11-8c94-487b-b67a-875da7b7be21-kube-api-access-tbwq6" (OuterVolumeSpecName: "kube-api-access-tbwq6") pod "5d939f11-8c94-487b-b67a-875da7b7be21" (UID: "5d939f11-8c94-487b-b67a-875da7b7be21"). InnerVolumeSpecName "kube-api-access-tbwq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:59:34.119768 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.119731 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d939f11-8c94-487b-b67a-875da7b7be21-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:34.119768 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.119760 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d939f11-8c94-487b-b67a-875da7b7be21-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:34.119768 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.119773 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbwq6\" (UniqueName: \"kubernetes.io/projected/5d939f11-8c94-487b-b67a-875da7b7be21-kube-api-access-tbwq6\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:34.391710 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.391617 2579 generic.go:356] "Generic (PLEG): container finished" podID="5d939f11-8c94-487b-b67a-875da7b7be21" containerID="e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578" exitCode=0 Sep 15 20:59:34.391710 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.391663 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" Sep 15 20:59:34.391710 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.391678 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" event={"ID":"5d939f11-8c94-487b-b67a-875da7b7be21","Type":"ContainerDied","Data":"e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578"} Sep 15 20:59:34.391964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.391719 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr" event={"ID":"5d939f11-8c94-487b-b67a-875da7b7be21","Type":"ContainerDied","Data":"20a32fc5ecdcf8967d1811a57e1e27e5676f29679c175bae9b487b707c07c748"} Sep 15 20:59:34.391964 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.391742 2579 scope.go:117] "RemoveContainer" containerID="e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578" Sep 15 20:59:34.401365 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.401346 2579 scope.go:117] "RemoveContainer" containerID="e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578" Sep 15 20:59:34.401638 ip-10-0-130-107 kubenswrapper[2579]: E0915 20:59:34.401621 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578\": container with ID starting with e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578 not found: ID does not exist" containerID="e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578" Sep 15 20:59:34.401697 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.401648 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578"} err="failed to get container status \"e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578\": rpc error: code = NotFound desc = could not find container \"e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578\": container with ID starting with e116484fda09fd1ce7ce1c26bf33ee8af55eeca406af7fc1beebdf5ca8d11578 not found: ID does not exist" Sep 15 20:59:34.411962 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.411940 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr"] Sep 15 20:59:34.417523 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:34.417485 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-41588d/explicit-model-41588d-kserve-6597b6f449-4bdsr"] Sep 15 20:59:35.312260 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:35.312224 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d939f11-8c94-487b-b67a-875da7b7be21" path="/var/lib/kubelet/pods/5d939f11-8c94-487b-b67a-875da7b7be21/volumes" Sep 15 20:59:36.317426 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:36.317393 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b"] Sep 15 20:59:36.317936 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:36.317672 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" containerID="cri-o://d25ed7a50c27601a5105cba3853f8f4e5a99561cc288456c7d311bbf8bde03ae" gracePeriod=30 Sep 15 20:59:37.060674 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.060642 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg"] Sep 15 20:59:37.061211 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.061196 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019a8674-9df0-4c04-9eb7-b7550244719d" containerName="revoke-keys" Sep 15 20:59:37.061264 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.061216 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="019a8674-9df0-4c04-9eb7-b7550244719d" containerName="revoke-keys" Sep 15 20:59:37.061264 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.061242 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d939f11-8c94-487b-b67a-875da7b7be21" containerName="main" Sep 15 20:59:37.061264 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.061250 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d939f11-8c94-487b-b67a-875da7b7be21" containerName="main" Sep 15 20:59:37.061374 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.061280 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a9ff674-d9c5-413b-9a08-79c5869aa7f1" containerName="payload-pre-processing" Sep 15 20:59:37.061374 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.061290 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a9ff674-d9c5-413b-9a08-79c5869aa7f1" containerName="payload-pre-processing" Sep 15 20:59:37.061437 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.061386 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d939f11-8c94-487b-b67a-875da7b7be21" containerName="main" Sep 15 20:59:37.061437 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.061400 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a9ff674-d9c5-413b-9a08-79c5869aa7f1" containerName="payload-pre-processing" Sep 15 20:59:37.061437 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.061412 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="019a8674-9df0-4c04-9eb7-b7550244719d" containerName="revoke-keys" Sep 15 20:59:37.066257 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.066236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.068375 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.068357 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-990e4a-openshift-default-dockercfg-5vs2h\"" Sep 15 20:59:37.076227 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.076203 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg"] Sep 15 20:59:37.252346 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.252304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16272c8e-eca2-4274-b8b4-023f43c19499-istiod-ca-cert\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.252565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.252388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-credential-socket\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.252565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.252420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16272c8e-eca2-4274-b8b4-023f43c19499-istio-podinfo\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.252565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.252439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-socket\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.252565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.252481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-envoy\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.252565 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.252552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-istio-token\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.252779 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.252577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-data\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.252779 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.252594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-certs\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.252779 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.252664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzn2m\" (UniqueName: \"kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-kube-api-access-wzn2m\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.353866 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.353783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-credential-socket\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.353866 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.353829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16272c8e-eca2-4274-b8b4-023f43c19499-istio-podinfo\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.353866 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.353851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-socket\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.354436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.353878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-envoy\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.354436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.353902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-istio-token\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.354436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.353927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-data\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.354436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.353953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-certs\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.354436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.354008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzn2m\" (UniqueName: \"kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-kube-api-access-wzn2m\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.354436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.354051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16272c8e-eca2-4274-b8b4-023f43c19499-istiod-ca-cert\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.354436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.354228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-credential-socket\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.354436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.354253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-socket\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.354436 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.354334 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-data\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.354975 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.354487 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-certs\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.355015 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.355000 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16272c8e-eca2-4274-b8b4-023f43c19499-istiod-ca-cert\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.356338 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.356321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-envoy\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.356435 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.356372 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16272c8e-eca2-4274-b8b4-023f43c19499-istio-podinfo\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.370104 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.370079 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzn2m\" (UniqueName: \"kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-kube-api-access-wzn2m\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.373692 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.373673 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-istio-token\") pod \"e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.378691 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.378675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:37.653681 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.653657 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg"] Sep 15 20:59:37.655654 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:59:37.655634 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16272c8e_eca2_4274_b8b4_023f43c19499.slice/crio-8c3918661d27e58ff02cc4b395599f1f5ed03df2dc6e5c0e75a6c152e6bc5a64 WatchSource:0}: Error finding container 8c3918661d27e58ff02cc4b395599f1f5ed03df2dc6e5c0e75a6c152e6bc5a64: Status 404 returned error can't find the container with id 8c3918661d27e58ff02cc4b395599f1f5ed03df2dc6e5c0e75a6c152e6bc5a64 Sep 15 20:59:37.657962 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.657926 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:59:37.658052 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.657990 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:59:37.658052 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:37.658020 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 20:59:38.355338 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.355302 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52"] Sep 15 20:59:38.358856 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.358839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.361049 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.361029 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"e2e-conflict-a-466b1a2b-kserve-self-signed-certs\"" Sep 15 20:59:38.366977 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.366954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52"] Sep 15 20:59:38.415588 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.415555 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" event={"ID":"16272c8e-eca2-4274-b8b4-023f43c19499","Type":"ContainerStarted","Data":"2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49"} Sep 15 20:59:38.415753 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.415608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" event={"ID":"16272c8e-eca2-4274-b8b4-023f43c19499","Type":"ContainerStarted","Data":"8c3918661d27e58ff02cc4b395599f1f5ed03df2dc6e5c0e75a6c152e6bc5a64"} Sep 15 20:59:38.434665 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.434614 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" podStartSLOduration=1.434597605 podStartE2EDuration="1.434597605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:38.434411842 +0000 UTC m=+1375.628365297" watchObservedRunningTime="2026-09-15 20:59:38.434597605 +0000 UTC m=+1375.628551059" Sep 15 20:59:38.465711 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.465687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-model-cache\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.465851 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.465716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-dshm\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.465851 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.465749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tls-certs\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.465940 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.465858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tmp-dir\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.465940 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.465903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzcrf\" (UniqueName: \"kubernetes.io/projected/d07773c9-4cb0-4541-aaca-e1ddb32e915a-kube-api-access-tzcrf\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.465940 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.465919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-home\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.567364 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.567329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-model-cache\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.567364 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.567363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-dshm\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.567669 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.567421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tls-certs\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.567784 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.567758 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-model-cache\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.567899 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.567878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tmp-dir\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.567967 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.567950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzcrf\" (UniqueName: \"kubernetes.io/projected/d07773c9-4cb0-4541-aaca-e1ddb32e915a-kube-api-access-tzcrf\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.568031 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.567978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-home\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.568194 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.568174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tmp-dir\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.568330 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.568310 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-home\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.569798 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.569778 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-dshm\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.570056 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.570037 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tls-certs\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.575470 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.575446 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzcrf\" (UniqueName: \"kubernetes.io/projected/d07773c9-4cb0-4541-aaca-e1ddb32e915a-kube-api-access-tzcrf\") pod \"e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.677048 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.676967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:38.956977 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:38.956951 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52"] Sep 15 20:59:38.959232 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:59:38.959205 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd07773c9_4cb0_4541_aaca_e1ddb32e915a.slice/crio-03d4173c62f8201c38acfa155e91386d84c0563ab1334577a7286ecef01d0c38 WatchSource:0}: Error finding container 03d4173c62f8201c38acfa155e91386d84c0563ab1334577a7286ecef01d0c38: Status 404 returned error can't find the container with id 03d4173c62f8201c38acfa155e91386d84c0563ab1334577a7286ecef01d0c38 Sep 15 20:59:39.379130 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:39.379099 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:39.384082 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:39.384057 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:39.422559 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:39.422527 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" event={"ID":"d07773c9-4cb0-4541-aaca-e1ddb32e915a","Type":"ContainerStarted","Data":"0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153"} Sep 15 20:59:39.422745 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:39.422571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" event={"ID":"d07773c9-4cb0-4541-aaca-e1ddb32e915a","Type":"ContainerStarted","Data":"03d4173c62f8201c38acfa155e91386d84c0563ab1334577a7286ecef01d0c38"} Sep 15 20:59:39.422745 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:39.422739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:39.423683 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:39.423667 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 20:59:39.444063 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:39.444017 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" podStartSLOduration=1.444001448 podStartE2EDuration="1.444001448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:39.442833476 +0000 UTC m=+1376.636786933" watchObservedRunningTime="2026-09-15 20:59:39.444001448 +0000 UTC m=+1376.637954971" Sep 15 20:59:41.436704 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.436659 2579 generic.go:356] "Generic (PLEG): container finished" podID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerID="d25ed7a50c27601a5105cba3853f8f4e5a99561cc288456c7d311bbf8bde03ae" exitCode=0 Sep 15 20:59:41.437094 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.436733 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" event={"ID":"828414c3-79de-4b80-8fc2-fca21628c9a1","Type":"ContainerDied","Data":"d25ed7a50c27601a5105cba3853f8f4e5a99561cc288456c7d311bbf8bde03ae"} Sep 15 20:59:41.437094 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.436811 2579 scope.go:117] "RemoveContainer" containerID="dc6100193aa2eb4a707d14e1850b647d388f219256d1e958acf4e3d3c46826f9" Sep 15 20:59:41.573358 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.573334 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:59:41.596559 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.596478 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-token\") pod \"828414c3-79de-4b80-8fc2-fca21628c9a1\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " Sep 15 20:59:41.596559 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.596555 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-socket\") pod \"828414c3-79de-4b80-8fc2-fca21628c9a1\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " Sep 15 20:59:41.596849 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.596665 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-podinfo\") pod \"828414c3-79de-4b80-8fc2-fca21628c9a1\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " Sep 15 20:59:41.596849 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.596714 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctdg6\" (UniqueName: \"kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-kube-api-access-ctdg6\") pod \"828414c3-79de-4b80-8fc2-fca21628c9a1\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " Sep 15 20:59:41.596849 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.596759 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-envoy\") pod \"828414c3-79de-4b80-8fc2-fca21628c9a1\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " Sep 15 20:59:41.596849 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.596794 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-credential-socket\") pod \"828414c3-79de-4b80-8fc2-fca21628c9a1\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " Sep 15 20:59:41.596849 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.596817 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828414c3-79de-4b80-8fc2-fca21628c9a1-istiod-ca-cert\") pod \"828414c3-79de-4b80-8fc2-fca21628c9a1\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " Sep 15 20:59:41.597138 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.596853 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "828414c3-79de-4b80-8fc2-fca21628c9a1" (UID: "828414c3-79de-4b80-8fc2-fca21628c9a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:41.597138 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.596859 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-certs\") pod \"828414c3-79de-4b80-8fc2-fca21628c9a1\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " Sep 15 20:59:41.597138 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.596918 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-data\") pod \"828414c3-79de-4b80-8fc2-fca21628c9a1\" (UID: \"828414c3-79de-4b80-8fc2-fca21628c9a1\") " Sep 15 20:59:41.597138 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.597050 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "828414c3-79de-4b80-8fc2-fca21628c9a1" (UID: "828414c3-79de-4b80-8fc2-fca21628c9a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:41.597138 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.597117 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "828414c3-79de-4b80-8fc2-fca21628c9a1" (UID: "828414c3-79de-4b80-8fc2-fca21628c9a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:41.597398 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.597221 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-data" (OuterVolumeSpecName: "istio-data") pod "828414c3-79de-4b80-8fc2-fca21628c9a1" (UID: "828414c3-79de-4b80-8fc2-fca21628c9a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:41.597398 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.597380 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:41.597524 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.597400 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:41.597524 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.597414 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:41.597524 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.597426 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:41.598064 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.598033 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828414c3-79de-4b80-8fc2-fca21628c9a1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "828414c3-79de-4b80-8fc2-fca21628c9a1" (UID: "828414c3-79de-4b80-8fc2-fca21628c9a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:59:41.599452 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.599422 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-token" (OuterVolumeSpecName: "istio-token") pod "828414c3-79de-4b80-8fc2-fca21628c9a1" (UID: "828414c3-79de-4b80-8fc2-fca21628c9a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:59:41.599755 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.599732 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-kube-api-access-ctdg6" (OuterVolumeSpecName: "kube-api-access-ctdg6") pod "828414c3-79de-4b80-8fc2-fca21628c9a1" (UID: "828414c3-79de-4b80-8fc2-fca21628c9a1"). InnerVolumeSpecName "kube-api-access-ctdg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:59:41.599951 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.599934 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "828414c3-79de-4b80-8fc2-fca21628c9a1" (UID: "828414c3-79de-4b80-8fc2-fca21628c9a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:41.600538 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.600522 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "828414c3-79de-4b80-8fc2-fca21628c9a1" (UID: "828414c3-79de-4b80-8fc2-fca21628c9a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 20:59:41.698323 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.698289 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdg6\" (UniqueName: \"kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-kube-api-access-ctdg6\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:41.698323 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.698319 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:41.698539 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.698343 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828414c3-79de-4b80-8fc2-fca21628c9a1-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:41.698539 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.698351 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:41.698539 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:41.698359 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828414c3-79de-4b80-8fc2-fca21628c9a1-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 20:59:42.443353 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:42.443323 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" Sep 15 20:59:42.443820 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:42.443367 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b" event={"ID":"828414c3-79de-4b80-8fc2-fca21628c9a1","Type":"ContainerDied","Data":"33b8c5075b954dcbfd677f0bf09eab371db8563a20d9a9fdc6764bb37de6a175"} Sep 15 20:59:42.443820 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:42.443430 2579 scope.go:117] "RemoveContainer" containerID="d25ed7a50c27601a5105cba3853f8f4e5a99561cc288456c7d311bbf8bde03ae" Sep 15 20:59:42.464201 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:42.464175 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b"] Sep 15 20:59:42.467765 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:42.467742 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-41588d-openshift-default-6b75c76bd8-fhd2b"] Sep 15 20:59:43.312651 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:43.312620 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" path="/var/lib/kubelet/pods/828414c3-79de-4b80-8fc2-fca21628c9a1/volumes" Sep 15 20:59:48.677394 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:48.677358 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:48.677926 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:48.677441 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:48.689864 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:48.689843 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:49.492885 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:49.492854 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 20:59:57.356194 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.356159 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv"] Sep 15 20:59:57.356596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.356543 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" Sep 15 20:59:57.356596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.356557 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" Sep 15 20:59:57.356678 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.356628 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" Sep 15 20:59:57.356678 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.356639 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" Sep 15 20:59:57.359783 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.359765 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.362790 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.362763 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-990e4a\"/\"openshift-service-ca.crt\"" Sep 15 20:59:57.362790 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.362781 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-990e4a\"/\"noten-model-990e4a-kserve-self-signed-certs\"" Sep 15 20:59:57.362997 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.362783 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-990e4a\"/\"kube-root-ca.crt\"" Sep 15 20:59:57.362997 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.362787 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-990e4a\"/\"default-dockercfg-dzczx\"" Sep 15 20:59:57.369757 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.369724 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv"] Sep 15 20:59:57.463152 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.463119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-model-cache\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.463327 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.463165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t87gw\" (UniqueName: \"kubernetes.io/projected/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-kube-api-access-t87gw\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.463327 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.463203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-dshm\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.463327 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.463237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tls-certs\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.463327 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.463297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-home\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.463523 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.463340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tmp-dir\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.564596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.564564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t87gw\" (UniqueName: \"kubernetes.io/projected/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-kube-api-access-t87gw\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.564596 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.564603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-dshm\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.564803 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.564628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tls-certs\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.564803 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.564650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-home\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.564803 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.564689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tmp-dir\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.564803 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.564774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-model-cache\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.565097 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.565076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-home\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.565175 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.565110 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-model-cache\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.565175 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.565153 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tmp-dir\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.567072 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.567049 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-dshm\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.567198 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.567182 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tls-certs\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.571666 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.571642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t87gw\" (UniqueName: \"kubernetes.io/projected/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-kube-api-access-t87gw\") pod \"noten-model-990e4a-kserve-755d88b9d7-kr8zv\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.669633 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.669552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 20:59:57.941886 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:57.941855 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv"] Sep 15 20:59:57.943558 ip-10-0-130-107 kubenswrapper[2579]: W0915 20:59:57.943533 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea82e5b7_cce8_4b00_b19c_a7b2b1974976.slice/crio-078ecd595e86617dc8d959c338d98e743e9c606cbeffddfde22104158644e7c9 WatchSource:0}: Error finding container 078ecd595e86617dc8d959c338d98e743e9c606cbeffddfde22104158644e7c9: Status 404 returned error can't find the container with id 078ecd595e86617dc8d959c338d98e743e9c606cbeffddfde22104158644e7c9 Sep 15 20:59:58.532095 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:58.532057 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" event={"ID":"ea82e5b7-cce8-4b00-b19c-a7b2b1974976","Type":"ContainerStarted","Data":"d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52"} Sep 15 20:59:58.532095 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:58.532100 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" event={"ID":"ea82e5b7-cce8-4b00-b19c-a7b2b1974976","Type":"ContainerStarted","Data":"078ecd595e86617dc8d959c338d98e743e9c606cbeffddfde22104158644e7c9"} Sep 15 20:59:58.548039 ip-10-0-130-107 kubenswrapper[2579]: I0915 20:59:58.547989 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" podStartSLOduration=1.5479726600000001 podStartE2EDuration="1.54797266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:58.547148956 +0000 UTC m=+1395.741102411" watchObservedRunningTime="2026-09-15 20:59:58.54797266 +0000 UTC m=+1395.741926115" Sep 15 21:00:00.136780 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.136749 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb"] Sep 15 21:00:00.137242 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.137173 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" Sep 15 21:00:00.137242 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.137185 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="828414c3-79de-4b80-8fc2-fca21628c9a1" containerName="istio-proxy" Sep 15 21:00:00.140475 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.140452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb" Sep 15 21:00:00.142166 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.142142 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr"] Sep 15 21:00:00.142742 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.142723 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:00:00.146048 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.146028 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl"] Sep 15 21:00:00.146333 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.146302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr" Sep 15 21:00:00.150710 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.150689 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825100-84zz7"] Sep 15 21:00:00.150846 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.150826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl" Sep 15 21:00:00.154138 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.154114 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb"] Sep 15 21:00:00.154379 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.154361 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825100-84zz7" Sep 15 21:00:00.154996 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.154972 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb" Sep 15 21:00:00.156005 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.155987 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr"] Sep 15 21:00:00.161945 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.161925 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl"] Sep 15 21:00:00.165548 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.165526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr" Sep 15 21:00:00.180668 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.180641 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825100-84zz7"] Sep 15 21:00:00.214538 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.213627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl" Sep 15 21:00:00.221806 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.221191 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825100-84zz7" Sep 15 21:00:00.444959 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.444934 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb"] Sep 15 21:00:00.447102 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:00:00.447069 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81436216_f5ed_4497_9727_525f1c1db457.slice/crio-5ea14cb64876be1df42199524c7da052880dc8d91984afb5309229b3b8bc970c WatchSource:0}: Error finding container 5ea14cb64876be1df42199524c7da052880dc8d91984afb5309229b3b8bc970c: Status 404 returned error can't find the container with id 5ea14cb64876be1df42199524c7da052880dc8d91984afb5309229b3b8bc970c Sep 15 21:00:00.460760 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.460649 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr"] Sep 15 21:00:00.463471 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:00:00.463441 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b539d6_a1fe_45a5_9d58_1b71cde16b4a.slice/crio-2a0babdde07eb8516a3004114d0f432db9ae265b25612c66310cbedea7e9fb6f WatchSource:0}: Error finding container 2a0babdde07eb8516a3004114d0f432db9ae265b25612c66310cbedea7e9fb6f: Status 404 returned error can't find the container with id 2a0babdde07eb8516a3004114d0f432db9ae265b25612c66310cbedea7e9fb6f Sep 15 21:00:00.524069 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.524033 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825100-84zz7"] Sep 15 21:00:00.526170 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:00:00.526142 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41af4c96_d2fa_45d4_9157_4ce7a7e83014.slice/crio-c9ea3fd66367d810357a1e6c81795de941479163a2e15db70f5376fe73d90d6e WatchSource:0}: Error finding container c9ea3fd66367d810357a1e6c81795de941479163a2e15db70f5376fe73d90d6e: Status 404 returned error can't find the container with id c9ea3fd66367d810357a1e6c81795de941479163a2e15db70f5376fe73d90d6e Sep 15 21:00:00.536284 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.536261 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl"] Sep 15 21:00:00.543584 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:00:00.543554 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e56f904_6b9a_40bc_8be8_18b61977ba94.slice/crio-d01ebf015703e8219856232af258cd37351cf69c5fd43c08ffd3a9aca46917a5 WatchSource:0}: Error finding container d01ebf015703e8219856232af258cd37351cf69c5fd43c08ffd3a9aca46917a5: Status 404 returned error can't find the container with id d01ebf015703e8219856232af258cd37351cf69c5fd43c08ffd3a9aca46917a5 Sep 15 21:00:00.548304 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.548281 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr" event={"ID":"c8b539d6-a1fe-45a5-9d58-1b71cde16b4a","Type":"ContainerStarted","Data":"2a0babdde07eb8516a3004114d0f432db9ae265b25612c66310cbedea7e9fb6f"} Sep 15 21:00:00.549787 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.549765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb" event={"ID":"81436216-f5ed-4497-9727-525f1c1db457","Type":"ContainerStarted","Data":"5ea14cb64876be1df42199524c7da052880dc8d91984afb5309229b3b8bc970c"} Sep 15 21:00:00.551388 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:00.551361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825100-84zz7" event={"ID":"41af4c96-d2fa-45d4-9157-4ce7a7e83014","Type":"ContainerStarted","Data":"c9ea3fd66367d810357a1e6c81795de941479163a2e15db70f5376fe73d90d6e"} Sep 15 21:00:01.559410 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:01.559377 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr" event={"ID":"c8b539d6-a1fe-45a5-9d58-1b71cde16b4a","Type":"ContainerStarted","Data":"ecb0e09b96da4b5022c07c6d5078d0454d94dd93081ca2802d3c1cb4dda7332c"} Sep 15 21:00:01.561027 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:01.561002 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl" event={"ID":"5e56f904-6b9a-40bc-8be8-18b61977ba94","Type":"ContainerStarted","Data":"7d5e7d6d149821732ddcfb0948097710d176bff2e041136154f89c2fa1cad85f"} Sep 15 21:00:01.561144 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:01.561050 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl" event={"ID":"5e56f904-6b9a-40bc-8be8-18b61977ba94","Type":"ContainerStarted","Data":"d01ebf015703e8219856232af258cd37351cf69c5fd43c08ffd3a9aca46917a5"} Sep 15 21:00:01.562952 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:01.562926 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb" event={"ID":"81436216-f5ed-4497-9727-525f1c1db457","Type":"ContainerStarted","Data":"6bf0a5c3540277447422f7b7b5a2c4e27bb7724cabccff8d2dae2d3a21d5f94f"} Sep 15 21:00:01.564539 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:01.564489 2579 generic.go:356] "Generic (PLEG): container finished" podID="41af4c96-d2fa-45d4-9157-4ce7a7e83014" containerID="9831e5eeafe3ab2592a36370b9c421f0814357f5dd372e0c7c6f19ec006900d3" exitCode=0 Sep 15 21:00:01.564637 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:01.564562 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825100-84zz7" event={"ID":"41af4c96-d2fa-45d4-9157-4ce7a7e83014","Type":"ContainerDied","Data":"9831e5eeafe3ab2592a36370b9c421f0814357f5dd372e0c7c6f19ec006900d3"} Sep 15 21:00:01.573655 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:01.573604 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr" podStartSLOduration=1.5735869820000001 podStartE2EDuration="1.573586982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:01.573392126 +0000 UTC m=+1398.767345580" watchObservedRunningTime="2026-09-15 21:00:01.573586982 +0000 UTC m=+1398.767540439" Sep 15 21:00:01.587877 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:01.587833 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl" podStartSLOduration=1.587818612 podStartE2EDuration="1.587818612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:01.586644121 +0000 UTC m=+1398.780597578" watchObservedRunningTime="2026-09-15 21:00:01.587818612 +0000 UTC m=+1398.781772067" Sep 15 21:00:01.614679 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:01.614634 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb" podStartSLOduration=1.614620229 podStartE2EDuration="1.614620229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:01.612783846 +0000 UTC m=+1398.806737300" watchObservedRunningTime="2026-09-15 21:00:01.614620229 +0000 UTC m=+1398.808573719" Sep 15 21:00:02.572277 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:02.572246 2579 generic.go:356] "Generic (PLEG): container finished" podID="c8b539d6-a1fe-45a5-9d58-1b71cde16b4a" containerID="ecb0e09b96da4b5022c07c6d5078d0454d94dd93081ca2802d3c1cb4dda7332c" exitCode=0 Sep 15 21:00:02.572684 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:02.572332 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr" event={"ID":"c8b539d6-a1fe-45a5-9d58-1b71cde16b4a","Type":"ContainerDied","Data":"ecb0e09b96da4b5022c07c6d5078d0454d94dd93081ca2802d3c1cb4dda7332c"} Sep 15 21:00:02.573929 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:02.573907 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e56f904-6b9a-40bc-8be8-18b61977ba94" containerID="7d5e7d6d149821732ddcfb0948097710d176bff2e041136154f89c2fa1cad85f" exitCode=0 Sep 15 21:00:02.574049 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:02.573989 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl" event={"ID":"5e56f904-6b9a-40bc-8be8-18b61977ba94","Type":"ContainerDied","Data":"7d5e7d6d149821732ddcfb0948097710d176bff2e041136154f89c2fa1cad85f"} Sep 15 21:00:02.575590 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:02.575571 2579 generic.go:356] "Generic (PLEG): container finished" podID="81436216-f5ed-4497-9727-525f1c1db457" containerID="6bf0a5c3540277447422f7b7b5a2c4e27bb7724cabccff8d2dae2d3a21d5f94f" exitCode=0 Sep 15 21:00:02.575688 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:02.575597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb" event={"ID":"81436216-f5ed-4497-9727-525f1c1db457","Type":"ContainerDied","Data":"6bf0a5c3540277447422f7b7b5a2c4e27bb7724cabccff8d2dae2d3a21d5f94f"} Sep 15 21:00:02.715150 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:02.715120 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825100-84zz7" Sep 15 21:00:03.583326 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:03.583243 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825100-84zz7" Sep 15 21:00:03.583326 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:03.583284 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825100-84zz7" event={"ID":"41af4c96-d2fa-45d4-9157-4ce7a7e83014","Type":"ContainerDied","Data":"c9ea3fd66367d810357a1e6c81795de941479163a2e15db70f5376fe73d90d6e"} Sep 15 21:00:03.583842 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:03.583329 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ea3fd66367d810357a1e6c81795de941479163a2e15db70f5376fe73d90d6e" Sep 15 21:00:03.727414 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:03.727388 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl" Sep 15 21:00:03.757821 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:03.757797 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb" Sep 15 21:00:03.795068 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:03.795045 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr" Sep 15 21:00:04.590572 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:04.590538 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr" Sep 15 21:00:04.590572 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:04.590551 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr" event={"ID":"c8b539d6-a1fe-45a5-9d58-1b71cde16b4a","Type":"ContainerDied","Data":"2a0babdde07eb8516a3004114d0f432db9ae265b25612c66310cbedea7e9fb6f"} Sep 15 21:00:04.591079 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:04.590600 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a0babdde07eb8516a3004114d0f432db9ae265b25612c66310cbedea7e9fb6f" Sep 15 21:00:04.591975 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:04.591959 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl" Sep 15 21:00:04.592098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:04.591978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl" event={"ID":"5e56f904-6b9a-40bc-8be8-18b61977ba94","Type":"ContainerDied","Data":"d01ebf015703e8219856232af258cd37351cf69c5fd43c08ffd3a9aca46917a5"} Sep 15 21:00:04.592098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:04.592016 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d01ebf015703e8219856232af258cd37351cf69c5fd43c08ffd3a9aca46917a5" Sep 15 21:00:04.593259 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:04.593243 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb" Sep 15 21:00:04.593259 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:04.593261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb" event={"ID":"81436216-f5ed-4497-9727-525f1c1db457","Type":"ContainerDied","Data":"5ea14cb64876be1df42199524c7da052880dc8d91984afb5309229b3b8bc970c"} Sep 15 21:00:04.593386 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:04.593300 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea14cb64876be1df42199524c7da052880dc8d91984afb5309229b3b8bc970c" Sep 15 21:00:06.310171 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310134 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh"] Sep 15 21:00:06.310606 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310591 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e56f904-6b9a-40bc-8be8-18b61977ba94" containerName="cleanup" Sep 15 21:00:06.310652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310608 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e56f904-6b9a-40bc-8be8-18b61977ba94" containerName="cleanup" Sep 15 21:00:06.310652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310630 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41af4c96-d2fa-45d4-9157-4ce7a7e83014" containerName="cleanup" Sep 15 21:00:06.310652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310636 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41af4c96-d2fa-45d4-9157-4ce7a7e83014" containerName="cleanup" Sep 15 21:00:06.310652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310642 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81436216-f5ed-4497-9727-525f1c1db457" containerName="cleanup" Sep 15 21:00:06.310652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310648 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="81436216-f5ed-4497-9727-525f1c1db457" containerName="cleanup" Sep 15 21:00:06.310818 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310659 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b539d6-a1fe-45a5-9d58-1b71cde16b4a" containerName="cleanup" Sep 15 21:00:06.310818 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310665 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b539d6-a1fe-45a5-9d58-1b71cde16b4a" containerName="cleanup" Sep 15 21:00:06.310818 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310733 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e56f904-6b9a-40bc-8be8-18b61977ba94" containerName="cleanup" Sep 15 21:00:06.310818 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310740 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41af4c96-d2fa-45d4-9157-4ce7a7e83014" containerName="cleanup" Sep 15 21:00:06.310818 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310745 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b539d6-a1fe-45a5-9d58-1b71cde16b4a" containerName="cleanup" Sep 15 21:00:06.310818 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.310752 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="81436216-f5ed-4497-9727-525f1c1db457" containerName="cleanup" Sep 15 21:00:06.313948 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.313929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:00:06.316562 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.316544 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-3825fc-dockercfg-z9b2g\"" Sep 15 21:00:06.317021 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.317004 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3825fc\"" Sep 15 21:00:06.324946 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.324921 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh"] Sep 15 21:00:06.461841 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.461809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d5xq\" (UniqueName: \"kubernetes.io/projected/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-kube-api-access-9d5xq\") pod \"payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh\" (UID: \"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:00:06.461986 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.461941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh\" (UID: \"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:00:06.562975 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.562886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d5xq\" (UniqueName: \"kubernetes.io/projected/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-kube-api-access-9d5xq\") pod \"payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh\" (UID: \"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:00:06.563125 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.562990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh\" (UID: \"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:00:06.563714 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.563696 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh\" (UID: \"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:00:06.570600 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.570578 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d5xq\" (UniqueName: \"kubernetes.io/projected/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-kube-api-access-9d5xq\") pod \"payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh\" (UID: \"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:00:06.626999 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.626970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:00:06.898646 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:06.898616 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh"] Sep 15 21:00:06.900722 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:00:06.900693 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcaabf4_f53b_4fbb_bbc3_0a5c72714f4f.slice/crio-a6bc3994e858cf50f66e3b1bddcddd93fc6ca7c868d1632e6ac20bcefbb871f3 WatchSource:0}: Error finding container a6bc3994e858cf50f66e3b1bddcddd93fc6ca7c868d1632e6ac20bcefbb871f3: Status 404 returned error can't find the container with id a6bc3994e858cf50f66e3b1bddcddd93fc6ca7c868d1632e6ac20bcefbb871f3 Sep 15 21:00:07.612529 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:07.612477 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" event={"ID":"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f","Type":"ContainerStarted","Data":"e87a97a655bd0d9b0a903f49c29bcef1f799c736584b2bd96adaba7beaf5d7cd"} Sep 15 21:00:07.613011 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:07.612548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" event={"ID":"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f","Type":"ContainerStarted","Data":"a6bc3994e858cf50f66e3b1bddcddd93fc6ca7c868d1632e6ac20bcefbb871f3"} Sep 15 21:00:07.613011 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:07.612626 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:00:07.628071 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:07.628021 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" podStartSLOduration=1.6280067539999998 podStartE2EDuration="1.628006754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:07.626925832 +0000 UTC m=+1404.820879287" watchObservedRunningTime="2026-09-15 21:00:07.628006754 +0000 UTC m=+1404.821960257" Sep 15 21:00:07.670316 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:07.670277 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 21:00:07.670457 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:07.670332 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 21:00:07.683653 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:07.683629 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 21:00:08.628589 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:08.628556 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 21:00:18.619700 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:18.619667 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:00:28.055387 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:28.055346 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv"] Sep 15 21:00:28.169974 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:28.169937 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg"] Sep 15 21:00:28.170223 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:28.170197 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" podUID="16272c8e-eca2-4274-b8b4-023f43c19499" containerName="istio-proxy" containerID="cri-o://2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49" gracePeriod=30 Sep 15 21:00:33.422945 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.422919 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 21:00:33.534352 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534325 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16272c8e-eca2-4274-b8b4-023f43c19499-istiod-ca-cert\") pod \"16272c8e-eca2-4274-b8b4-023f43c19499\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " Sep 15 21:00:33.534560 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534382 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-credential-socket\") pod \"16272c8e-eca2-4274-b8b4-023f43c19499\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " Sep 15 21:00:33.534560 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534405 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-data\") pod \"16272c8e-eca2-4274-b8b4-023f43c19499\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " Sep 15 21:00:33.534560 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534436 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-envoy\") pod \"16272c8e-eca2-4274-b8b4-023f43c19499\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " Sep 15 21:00:33.534560 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534475 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-istio-token\") pod \"16272c8e-eca2-4274-b8b4-023f43c19499\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " Sep 15 21:00:33.534560 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534536 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16272c8e-eca2-4274-b8b4-023f43c19499-istio-podinfo\") pod \"16272c8e-eca2-4274-b8b4-023f43c19499\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " Sep 15 21:00:33.534839 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534566 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-certs\") pod \"16272c8e-eca2-4274-b8b4-023f43c19499\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " Sep 15 21:00:33.534839 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534626 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzn2m\" (UniqueName: \"kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-kube-api-access-wzn2m\") pod \"16272c8e-eca2-4274-b8b4-023f43c19499\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " Sep 15 21:00:33.534839 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534656 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-socket\") pod \"16272c8e-eca2-4274-b8b4-023f43c19499\" (UID: \"16272c8e-eca2-4274-b8b4-023f43c19499\") " Sep 15 21:00:33.535004 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534832 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-data" (OuterVolumeSpecName: "istio-data") pod "16272c8e-eca2-4274-b8b4-023f43c19499" (UID: "16272c8e-eca2-4274-b8b4-023f43c19499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:00:33.535004 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534864 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "16272c8e-eca2-4274-b8b4-023f43c19499" (UID: "16272c8e-eca2-4274-b8b4-023f43c19499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:00:33.535004 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534885 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16272c8e-eca2-4274-b8b4-023f43c19499-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "16272c8e-eca2-4274-b8b4-023f43c19499" (UID: "16272c8e-eca2-4274-b8b4-023f43c19499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:00:33.535004 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.534944 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "16272c8e-eca2-4274-b8b4-023f43c19499" (UID: "16272c8e-eca2-4274-b8b4-023f43c19499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:00:33.535203 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.535057 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "16272c8e-eca2-4274-b8b4-023f43c19499" (UID: "16272c8e-eca2-4274-b8b4-023f43c19499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:00:33.535203 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.535119 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16272c8e-eca2-4274-b8b4-023f43c19499-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:33.535203 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.535133 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:33.535203 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.535142 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:33.535203 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.535151 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:33.536929 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.536906 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "16272c8e-eca2-4274-b8b4-023f43c19499" (UID: "16272c8e-eca2-4274-b8b4-023f43c19499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:00:33.537035 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.536941 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-kube-api-access-wzn2m" (OuterVolumeSpecName: "kube-api-access-wzn2m") pod "16272c8e-eca2-4274-b8b4-023f43c19499" (UID: "16272c8e-eca2-4274-b8b4-023f43c19499"). InnerVolumeSpecName "kube-api-access-wzn2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:00:33.537194 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.537170 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-istio-token" (OuterVolumeSpecName: "istio-token") pod "16272c8e-eca2-4274-b8b4-023f43c19499" (UID: "16272c8e-eca2-4274-b8b4-023f43c19499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:00:33.537329 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.537186 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16272c8e-eca2-4274-b8b4-023f43c19499-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "16272c8e-eca2-4274-b8b4-023f43c19499" (UID: "16272c8e-eca2-4274-b8b4-023f43c19499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:00:33.635739 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.635703 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzn2m\" (UniqueName: \"kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-kube-api-access-wzn2m\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:33.635739 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.635736 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:33.635997 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.635748 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16272c8e-eca2-4274-b8b4-023f43c19499-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:33.635997 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.635758 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16272c8e-eca2-4274-b8b4-023f43c19499-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:33.635997 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.635768 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16272c8e-eca2-4274-b8b4-023f43c19499-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:33.762870 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.762835 2579 generic.go:356] "Generic (PLEG): container finished" podID="16272c8e-eca2-4274-b8b4-023f43c19499" containerID="2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49" exitCode=0 Sep 15 21:00:33.763056 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.762885 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" Sep 15 21:00:33.763056 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.762908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" event={"ID":"16272c8e-eca2-4274-b8b4-023f43c19499","Type":"ContainerDied","Data":"2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49"} Sep 15 21:00:33.763056 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.762963 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg" event={"ID":"16272c8e-eca2-4274-b8b4-023f43c19499","Type":"ContainerDied","Data":"8c3918661d27e58ff02cc4b395599f1f5ed03df2dc6e5c0e75a6c152e6bc5a64"} Sep 15 21:00:33.763056 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.762992 2579 scope.go:117] "RemoveContainer" containerID="2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49" Sep 15 21:00:33.772991 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.772970 2579 scope.go:117] "RemoveContainer" containerID="2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49" Sep 15 21:00:33.773276 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:00:33.773256 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49\": container with ID starting with 2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49 not found: ID does not exist" containerID="2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49" Sep 15 21:00:33.773334 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.773289 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49"} err="failed to get container status \"2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49\": rpc error: code = NotFound desc = could not find container \"2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49\": container with ID starting with 2bc827ba43f3344242fd0294cb8eecff3bf6dd0b9ed353cd745323a2f6a88e49 not found: ID does not exist" Sep 15 21:00:33.791198 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.791121 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg"] Sep 15 21:00:33.792423 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:33.792398 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-990e4a-openshift-default-7c5996675b-5bnmg"] Sep 15 21:00:35.312532 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:35.312431 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16272c8e-eca2-4274-b8b4-023f43c19499" path="/var/lib/kubelet/pods/16272c8e-eca2-4274-b8b4-023f43c19499/volumes" Sep 15 21:00:41.655809 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.655768 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g"] Sep 15 21:00:41.656219 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.656165 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16272c8e-eca2-4274-b8b4-023f43c19499" containerName="istio-proxy" Sep 15 21:00:41.656219 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.656177 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16272c8e-eca2-4274-b8b4-023f43c19499" containerName="istio-proxy" Sep 15 21:00:41.656348 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.656233 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16272c8e-eca2-4274-b8b4-023f43c19499" containerName="istio-proxy" Sep 15 21:00:41.660973 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.660944 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.663307 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.663286 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-65001159-openshift-default-dockercfg-xxvzs\"" Sep 15 21:00:41.671049 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.670876 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g"] Sep 15 21:00:41.818241 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.818203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-envoy\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.818241 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.818242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-socket\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.818467 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.818263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-token\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.818467 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.818316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-podinfo\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.818467 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.818342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istiod-ca-cert\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.818467 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.818431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-data\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.818641 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.818470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jg8\" (UniqueName: \"kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-kube-api-access-v8jg8\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.818641 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.818524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-credential-socket\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.818641 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.818597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-certs\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-data\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jg8\" (UniqueName: \"kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-kube-api-access-v8jg8\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919370 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-credential-socket\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919370 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-certs\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919474 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-envoy\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919474 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-socket\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919626 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-token\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919626 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-podinfo\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919626 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istiod-ca-cert\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919626 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919604 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-data\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919838 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-certs\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919838 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919815 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-socket\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.919954 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.919924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-credential-socket\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.920400 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.920378 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istiod-ca-cert\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.921859 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.921841 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-envoy\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.922170 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.922148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-podinfo\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.927775 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.927743 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jg8\" (UniqueName: \"kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-kube-api-access-v8jg8\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.927874 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.927749 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-token\") pod \"e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:41.983945 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:41.983911 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:42.260961 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:42.260933 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g"] Sep 15 21:00:42.263002 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:00:42.262974 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab52b61_ddc4_457a_9ffd_7a895e3ab9e7.slice/crio-39939df3002fc1caa629caf6bada99733df83c2900450d869ee59e8c4e26e49a WatchSource:0}: Error finding container 39939df3002fc1caa629caf6bada99733df83c2900450d869ee59e8c4e26e49a: Status 404 returned error can't find the container with id 39939df3002fc1caa629caf6bada99733df83c2900450d869ee59e8c4e26e49a Sep 15 21:00:42.265734 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:42.265699 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:00:42.265831 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:42.265785 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:00:42.265895 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:42.265827 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:00:42.820775 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:42.820738 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" event={"ID":"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7","Type":"ContainerStarted","Data":"9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263"} Sep 15 21:00:42.821234 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:42.820795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" event={"ID":"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7","Type":"ContainerStarted","Data":"39939df3002fc1caa629caf6bada99733df83c2900450d869ee59e8c4e26e49a"} Sep 15 21:00:42.840390 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:42.840336 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" podStartSLOduration=1.8403172620000001 podStartE2EDuration="1.840317262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:42.839127474 +0000 UTC m=+1440.033080930" watchObservedRunningTime="2026-09-15 21:00:42.840317262 +0000 UTC m=+1440.034270720" Sep 15 21:00:42.984733 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:42.984700 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:43.075406 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.075319 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" podUID="ea82e5b7-cce8-4b00-b19c-a7b2b1974976" containerName="main" containerID="cri-o://d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52" gracePeriod=45 Sep 15 21:00:43.334367 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.334296 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 21:00:43.433740 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.433710 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tls-certs\") pod \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " Sep 15 21:00:43.433905 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.433785 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tmp-dir\") pod \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " Sep 15 21:00:43.433905 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.433804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-home\") pod \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " Sep 15 21:00:43.433905 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.433841 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-dshm\") pod \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " Sep 15 21:00:43.433905 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.433892 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-model-cache\") pod \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " Sep 15 21:00:43.434130 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.433947 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t87gw\" (UniqueName: \"kubernetes.io/projected/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-kube-api-access-t87gw\") pod \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\" (UID: \"ea82e5b7-cce8-4b00-b19c-a7b2b1974976\") " Sep 15 21:00:43.434130 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.434069 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-home" (OuterVolumeSpecName: "home") pod "ea82e5b7-cce8-4b00-b19c-a7b2b1974976" (UID: "ea82e5b7-cce8-4b00-b19c-a7b2b1974976"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:00:43.434130 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.434080 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea82e5b7-cce8-4b00-b19c-a7b2b1974976" (UID: "ea82e5b7-cce8-4b00-b19c-a7b2b1974976"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:00:43.434288 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.434193 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea82e5b7-cce8-4b00-b19c-a7b2b1974976" (UID: "ea82e5b7-cce8-4b00-b19c-a7b2b1974976"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:00:43.434346 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.434301 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:43.434346 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.434314 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:43.434346 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.434322 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:43.434747 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.434723 2579 scope.go:117] "RemoveContainer" containerID="d7f7241e017c0acf4c1d5c53853ce4b7ae0523a6b8bb21f36572ec5aa842fbe5" Sep 15 21:00:43.436436 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.436400 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea82e5b7-cce8-4b00-b19c-a7b2b1974976" (UID: "ea82e5b7-cce8-4b00-b19c-a7b2b1974976"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:00:43.436544 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.436450 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-kube-api-access-t87gw" (OuterVolumeSpecName: "kube-api-access-t87gw") pod "ea82e5b7-cce8-4b00-b19c-a7b2b1974976" (UID: "ea82e5b7-cce8-4b00-b19c-a7b2b1974976"). InnerVolumeSpecName "kube-api-access-t87gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:00:43.436544 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.436476 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-dshm" (OuterVolumeSpecName: "dshm") pod "ea82e5b7-cce8-4b00-b19c-a7b2b1974976" (UID: "ea82e5b7-cce8-4b00-b19c-a7b2b1974976"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:00:43.535445 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.535413 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:43.535445 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.535438 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t87gw\" (UniqueName: \"kubernetes.io/projected/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-kube-api-access-t87gw\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:43.535445 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.535450 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea82e5b7-cce8-4b00-b19c-a7b2b1974976-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:00:43.828594 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.828555 2579 generic.go:356] "Generic (PLEG): container finished" podID="ea82e5b7-cce8-4b00-b19c-a7b2b1974976" containerID="d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52" exitCode=0 Sep 15 21:00:43.829020 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.828607 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" Sep 15 21:00:43.829020 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.828645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" event={"ID":"ea82e5b7-cce8-4b00-b19c-a7b2b1974976","Type":"ContainerDied","Data":"d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52"} Sep 15 21:00:43.829020 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.828692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv" event={"ID":"ea82e5b7-cce8-4b00-b19c-a7b2b1974976","Type":"ContainerDied","Data":"078ecd595e86617dc8d959c338d98e743e9c606cbeffddfde22104158644e7c9"} Sep 15 21:00:43.829020 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.828712 2579 scope.go:117] "RemoveContainer" containerID="d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52" Sep 15 21:00:43.840837 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.840476 2579 scope.go:117] "RemoveContainer" containerID="d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52" Sep 15 21:00:43.840939 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:00:43.840907 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52\": container with ID starting with d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52 not found: ID does not exist" containerID="d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52" Sep 15 21:00:43.840997 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.840946 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52"} err="failed to get container status \"d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52\": rpc error: code = NotFound desc = could not find container \"d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52\": container with ID starting with d715fc9a9c7acf58df148f80311e4a16a20b7ee74b62ab46f1bdb1a7e3f3db52 not found: ID does not exist" Sep 15 21:00:43.856263 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.856235 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv"] Sep 15 21:00:43.859277 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.859243 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-990e4a/noten-model-990e4a-kserve-755d88b9d7-kr8zv"] Sep 15 21:00:43.990353 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:43.990324 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:44.835656 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:44.835620 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:44.836544 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:44.836526 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:00:45.311697 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:45.311669 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea82e5b7-cce8-4b00-b19c-a7b2b1974976" path="/var/lib/kubelet/pods/ea82e5b7-cce8-4b00-b19c-a7b2b1974976/volumes" Sep 15 21:00:49.866640 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.866604 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp"] Sep 15 21:00:49.867732 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.867709 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea82e5b7-cce8-4b00-b19c-a7b2b1974976" containerName="main" Sep 15 21:00:49.867732 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.867734 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea82e5b7-cce8-4b00-b19c-a7b2b1974976" containerName="main" Sep 15 21:00:49.867948 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.867935 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea82e5b7-cce8-4b00-b19c-a7b2b1974976" containerName="main" Sep 15 21:00:49.876364 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.876342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:00:49.880555 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.880530 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-65001159\"" Sep 15 21:00:49.880555 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.880536 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-65001159-dockercfg-kj4n8\"" Sep 15 21:00:49.885185 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.884519 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp"] Sep 15 21:00:49.894201 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.894178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86e6587a-a3c8-4352-8871-7ae10c1e7e58-plugins-config-volume\") pod \"payload-processing-e2e-mt-65001159-dc75856b9-nq6hp\" (UID: \"86e6587a-a3c8-4352-8871-7ae10c1e7e58\") " pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:00:49.894302 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.894278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgrv\" (UniqueName: \"kubernetes.io/projected/86e6587a-a3c8-4352-8871-7ae10c1e7e58-kube-api-access-8qgrv\") pod \"payload-processing-e2e-mt-65001159-dc75856b9-nq6hp\" (UID: \"86e6587a-a3c8-4352-8871-7ae10c1e7e58\") " pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:00:49.994937 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.994901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgrv\" (UniqueName: \"kubernetes.io/projected/86e6587a-a3c8-4352-8871-7ae10c1e7e58-kube-api-access-8qgrv\") pod \"payload-processing-e2e-mt-65001159-dc75856b9-nq6hp\" (UID: \"86e6587a-a3c8-4352-8871-7ae10c1e7e58\") " pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:00:49.995152 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.994973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86e6587a-a3c8-4352-8871-7ae10c1e7e58-plugins-config-volume\") pod \"payload-processing-e2e-mt-65001159-dc75856b9-nq6hp\" (UID: \"86e6587a-a3c8-4352-8871-7ae10c1e7e58\") " pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:00:49.995751 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:49.995733 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86e6587a-a3c8-4352-8871-7ae10c1e7e58-plugins-config-volume\") pod \"payload-processing-e2e-mt-65001159-dc75856b9-nq6hp\" (UID: \"86e6587a-a3c8-4352-8871-7ae10c1e7e58\") " pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:00:50.005826 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:50.005790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgrv\" (UniqueName: \"kubernetes.io/projected/86e6587a-a3c8-4352-8871-7ae10c1e7e58-kube-api-access-8qgrv\") pod \"payload-processing-e2e-mt-65001159-dc75856b9-nq6hp\" (UID: \"86e6587a-a3c8-4352-8871-7ae10c1e7e58\") " pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:00:50.189900 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:50.189819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:00:50.475725 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:50.475690 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp"] Sep 15 21:00:50.478408 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:00:50.478378 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e6587a_a3c8_4352_8871_7ae10c1e7e58.slice/crio-9d7a8c7eec7a372a3a165ae8f5501ccd1e05344357606967dff6454bcdd2c90b WatchSource:0}: Error finding container 9d7a8c7eec7a372a3a165ae8f5501ccd1e05344357606967dff6454bcdd2c90b: Status 404 returned error can't find the container with id 9d7a8c7eec7a372a3a165ae8f5501ccd1e05344357606967dff6454bcdd2c90b Sep 15 21:00:50.870294 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:50.870256 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" event={"ID":"86e6587a-a3c8-4352-8871-7ae10c1e7e58","Type":"ContainerStarted","Data":"b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d"} Sep 15 21:00:50.870790 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:50.870321 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" event={"ID":"86e6587a-a3c8-4352-8871-7ae10c1e7e58","Type":"ContainerStarted","Data":"9d7a8c7eec7a372a3a165ae8f5501ccd1e05344357606967dff6454bcdd2c90b"} Sep 15 21:00:50.870790 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:50.870440 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:00:50.891971 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:00:50.891918 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" podStartSLOduration=1.8919004350000002 podStartE2EDuration="1.891900435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:50.889266713 +0000 UTC m=+1448.083220169" watchObservedRunningTime="2026-09-15 21:00:50.891900435 +0000 UTC m=+1448.085853891" Sep 15 21:01:00.164484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.164455 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw"] Sep 15 21:01:00.168160 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.168139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.171184 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.171162 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"openshift-service-ca.crt\"" Sep 15 21:01:00.171292 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.171207 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"default-dockercfg-qswjz\"" Sep 15 21:01:00.171292 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.171252 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"distinct-w0-3f1c93-kserve-self-signed-certs\"" Sep 15 21:01:00.171292 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.171207 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"kube-root-ca.crt\"" Sep 15 21:01:00.176171 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.176145 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw"] Sep 15 21:01:00.292939 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.292905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-tmp-dir\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.293085 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.292959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fk2r\" (UniqueName: \"kubernetes.io/projected/199b0526-668c-4746-8455-dcb606fd015b-kube-api-access-4fk2r\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.293085 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.293002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-dshm\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.293085 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.293027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-model-cache\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.293211 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.293086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-home\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.293211 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.293121 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b0526-668c-4746-8455-dcb606fd015b-tls-certs\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.394189 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.394156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-home\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.394189 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.394191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b0526-668c-4746-8455-dcb606fd015b-tls-certs\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.394410 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.394241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-tmp-dir\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.394410 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.394276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fk2r\" (UniqueName: \"kubernetes.io/projected/199b0526-668c-4746-8455-dcb606fd015b-kube-api-access-4fk2r\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.394410 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.394298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-dshm\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.394410 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.394327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-model-cache\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.394661 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.394561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-home\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.394661 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.394640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-tmp-dir\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.394749 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.394684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-model-cache\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.396609 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.396589 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-dshm\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.396724 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.396703 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b0526-668c-4746-8455-dcb606fd015b-tls-certs\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.404036 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.404009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fk2r\" (UniqueName: \"kubernetes.io/projected/199b0526-668c-4746-8455-dcb606fd015b-kube-api-access-4fk2r\") pod \"distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.479168 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.479093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:00.770245 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.770217 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw"] Sep 15 21:01:00.928151 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.928110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" event={"ID":"199b0526-668c-4746-8455-dcb606fd015b","Type":"ContainerStarted","Data":"e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863"} Sep 15 21:01:00.928313 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.928162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" event={"ID":"199b0526-668c-4746-8455-dcb606fd015b","Type":"ContainerStarted","Data":"82142bba388648c8a31d69b703742654200926765b0f9efa8250c544beae9a10"} Sep 15 21:01:00.948462 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.948415 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" podStartSLOduration=0.948398174 podStartE2EDuration="948.398174ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:01:00.946973458 +0000 UTC m=+1458.140926913" watchObservedRunningTime="2026-09-15 21:01:00.948398174 +0000 UTC m=+1458.142351629" Sep 15 21:01:00.966135 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.966103 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm"] Sep 15 21:01:00.969714 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.969695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" Sep 15 21:01:00.971846 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.971829 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:01:00.976595 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:00.976568 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm"] Sep 15 21:01:01.101983 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:01.101939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0c7f3c-6ea8-4a09-b721-ea43d598551e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm\" (UID: \"9e0c7f3c-6ea8-4a09-b721-ea43d598551e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" Sep 15 21:01:01.203510 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:01.203472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0c7f3c-6ea8-4a09-b721-ea43d598551e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm\" (UID: \"9e0c7f3c-6ea8-4a09-b721-ea43d598551e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" Sep 15 21:01:01.204091 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:01.204070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0c7f3c-6ea8-4a09-b721-ea43d598551e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm\" (UID: \"9e0c7f3c-6ea8-4a09-b721-ea43d598551e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" Sep 15 21:01:01.282192 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:01.282161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" Sep 15 21:01:01.557327 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:01.557299 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm"] Sep 15 21:01:01.559167 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:01:01.559131 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0c7f3c_6ea8_4a09_b721_ea43d598551e.slice/crio-da0e5e34d690deafb2c3cb73f7546e08e8bf8c4c81fde21357458cc3047530b9 WatchSource:0}: Error finding container da0e5e34d690deafb2c3cb73f7546e08e8bf8c4c81fde21357458cc3047530b9: Status 404 returned error can't find the container with id da0e5e34d690deafb2c3cb73f7546e08e8bf8c4c81fde21357458cc3047530b9 Sep 15 21:01:01.879471 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:01.879446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:01:01.938242 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:01.938211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" event={"ID":"9e0c7f3c-6ea8-4a09-b721-ea43d598551e","Type":"ContainerStarted","Data":"e5e5370eca8be8542fa570e70e880f6236df4112c52c849815e18d01fa7ce26f"} Sep 15 21:01:01.938403 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:01.938272 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" event={"ID":"9e0c7f3c-6ea8-4a09-b721-ea43d598551e","Type":"ContainerStarted","Data":"da0e5e34d690deafb2c3cb73f7546e08e8bf8c4c81fde21357458cc3047530b9"} Sep 15 21:01:01.953253 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:01.953196 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" podStartSLOduration=1.953181236 podStartE2EDuration="1.953181236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:01:01.951990017 +0000 UTC m=+1459.145943474" watchObservedRunningTime="2026-09-15 21:01:01.953181236 +0000 UTC m=+1459.147134690" Sep 15 21:01:02.945737 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:02.945702 2579 generic.go:356] "Generic (PLEG): container finished" podID="9e0c7f3c-6ea8-4a09-b721-ea43d598551e" containerID="e5e5370eca8be8542fa570e70e880f6236df4112c52c849815e18d01fa7ce26f" exitCode=0 Sep 15 21:01:02.946141 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:02.945737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" event={"ID":"9e0c7f3c-6ea8-4a09-b721-ea43d598551e","Type":"ContainerDied","Data":"e5e5370eca8be8542fa570e70e880f6236df4112c52c849815e18d01fa7ce26f"} Sep 15 21:01:04.086987 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:04.086958 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" Sep 15 21:01:04.233899 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:04.233817 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0c7f3c-6ea8-4a09-b721-ea43d598551e-maas-api-service-ca\") pod \"9e0c7f3c-6ea8-4a09-b721-ea43d598551e\" (UID: \"9e0c7f3c-6ea8-4a09-b721-ea43d598551e\") " Sep 15 21:01:04.234187 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:04.234165 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0c7f3c-6ea8-4a09-b721-ea43d598551e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e0c7f3c-6ea8-4a09-b721-ea43d598551e" (UID: "9e0c7f3c-6ea8-4a09-b721-ea43d598551e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:01:04.334895 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:04.334859 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0c7f3c-6ea8-4a09-b721-ea43d598551e-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:04.961227 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:04.961194 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" Sep 15 21:01:04.961406 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:04.961225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm" event={"ID":"9e0c7f3c-6ea8-4a09-b721-ea43d598551e","Type":"ContainerDied","Data":"da0e5e34d690deafb2c3cb73f7546e08e8bf8c4c81fde21357458cc3047530b9"} Sep 15 21:01:04.961406 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:04.961305 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da0e5e34d690deafb2c3cb73f7546e08e8bf8c4c81fde21357458cc3047530b9" Sep 15 21:01:10.479274 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:10.479230 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:10.480435 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:10.480407 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:10.510014 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:10.509990 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:11.020400 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:11.020368 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:01:12.807271 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:12.807234 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk"] Sep 15 21:01:12.808044 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:12.808013 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" podUID="7a9100c0-11b2-49f0-acb0-10cad430ee72" containerName="payload-processing" containerID="cri-o://44bfc5b2c08d2c104f4617327b10b4374e7bf95a317b2e9f1e86b9bf1b2d22e1" gracePeriod=30 Sep 15 21:01:12.956971 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:12.956929 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm"] Sep 15 21:01:12.961279 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:12.961249 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5b5b9-lf9qm"] Sep 15 21:01:13.022389 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:13.022344 2579 generic.go:356] "Generic (PLEG): container finished" podID="7a9100c0-11b2-49f0-acb0-10cad430ee72" containerID="44bfc5b2c08d2c104f4617327b10b4374e7bf95a317b2e9f1e86b9bf1b2d22e1" exitCode=0 Sep 15 21:01:13.022389 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:13.022385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" event={"ID":"7a9100c0-11b2-49f0-acb0-10cad430ee72","Type":"ContainerDied","Data":"44bfc5b2c08d2c104f4617327b10b4374e7bf95a317b2e9f1e86b9bf1b2d22e1"} Sep 15 21:01:13.108459 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:13.108436 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 21:01:13.231333 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:13.231287 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9100c0-11b2-49f0-acb0-10cad430ee72-plugins-config-volume\") pod \"7a9100c0-11b2-49f0-acb0-10cad430ee72\" (UID: \"7a9100c0-11b2-49f0-acb0-10cad430ee72\") " Sep 15 21:01:13.231540 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:13.231442 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llv2k\" (UniqueName: \"kubernetes.io/projected/7a9100c0-11b2-49f0-acb0-10cad430ee72-kube-api-access-llv2k\") pod \"7a9100c0-11b2-49f0-acb0-10cad430ee72\" (UID: \"7a9100c0-11b2-49f0-acb0-10cad430ee72\") " Sep 15 21:01:13.231668 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:13.231642 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9100c0-11b2-49f0-acb0-10cad430ee72-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a9100c0-11b2-49f0-acb0-10cad430ee72" (UID: "7a9100c0-11b2-49f0-acb0-10cad430ee72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:01:13.231740 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:13.231728 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9100c0-11b2-49f0-acb0-10cad430ee72-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:13.233758 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:13.233737 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9100c0-11b2-49f0-acb0-10cad430ee72-kube-api-access-llv2k" (OuterVolumeSpecName: "kube-api-access-llv2k") pod "7a9100c0-11b2-49f0-acb0-10cad430ee72" (UID: "7a9100c0-11b2-49f0-acb0-10cad430ee72"). InnerVolumeSpecName "kube-api-access-llv2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:01:13.311553 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:13.311487 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e0c7f3c-6ea8-4a09-b721-ea43d598551e" path="/var/lib/kubelet/pods/9e0c7f3c-6ea8-4a09-b721-ea43d598551e/volumes" Sep 15 21:01:13.332766 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:13.332738 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llv2k\" (UniqueName: \"kubernetes.io/projected/7a9100c0-11b2-49f0-acb0-10cad430ee72-kube-api-access-llv2k\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:14.029915 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:14.029883 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" Sep 15 21:01:14.030342 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:14.029916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk" event={"ID":"7a9100c0-11b2-49f0-acb0-10cad430ee72","Type":"ContainerDied","Data":"6d5864ac5e77ebbfca40c4fc3b5d0ede54e2dd717c9b4fe6a200b9f5acea3c11"} Sep 15 21:01:14.030342 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:14.029978 2579 scope.go:117] "RemoveContainer" containerID="44bfc5b2c08d2c104f4617327b10b4374e7bf95a317b2e9f1e86b9bf1b2d22e1" Sep 15 21:01:14.051371 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:14.051343 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk"] Sep 15 21:01:14.059378 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:14.059354 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5b5b9-74ff4cb8ff-r2wrk"] Sep 15 21:01:15.313563 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:15.313527 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a9100c0-11b2-49f0-acb0-10cad430ee72" path="/var/lib/kubelet/pods/7a9100c0-11b2-49f0-acb0-10cad430ee72/volumes" Sep 15 21:01:19.060337 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.060306 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerID="9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263" exitCode=0 Sep 15 21:01:19.060803 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.060343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" event={"ID":"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7","Type":"ContainerDied","Data":"9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263"} Sep 15 21:01:19.060803 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.060766 2579 scope.go:117] "RemoveContainer" containerID="9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263" Sep 15 21:01:19.062029 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.061999 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:01:19.062103 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.062072 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:01:19.062140 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.062104 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:01:19.767877 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.767839 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6"] Sep 15 21:01:19.768262 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.768249 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a9100c0-11b2-49f0-acb0-10cad430ee72" containerName="payload-processing" Sep 15 21:01:19.768313 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.768264 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a9100c0-11b2-49f0-acb0-10cad430ee72" containerName="payload-processing" Sep 15 21:01:19.768313 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.768276 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0c7f3c-6ea8-4a09-b721-ea43d598551e" containerName="revoke-keys" Sep 15 21:01:19.768313 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.768283 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0c7f3c-6ea8-4a09-b721-ea43d598551e" containerName="revoke-keys" Sep 15 21:01:19.768411 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.768375 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a9100c0-11b2-49f0-acb0-10cad430ee72" containerName="payload-processing" Sep 15 21:01:19.768411 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.768384 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e0c7f3c-6ea8-4a09-b721-ea43d598551e" containerName="revoke-keys" Sep 15 21:01:19.771363 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.771347 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" Sep 15 21:01:19.773624 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.773602 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:01:19.778671 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.778647 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6"] Sep 15 21:01:19.795954 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.795926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/970e2add-359e-4476-beab-786fc59cbd27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6\" (UID: \"970e2add-359e-4476-beab-786fc59cbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" Sep 15 21:01:19.896582 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.896544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/970e2add-359e-4476-beab-786fc59cbd27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6\" (UID: \"970e2add-359e-4476-beab-786fc59cbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" Sep 15 21:01:19.897149 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:19.897129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/970e2add-359e-4476-beab-786fc59cbd27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6\" (UID: \"970e2add-359e-4476-beab-786fc59cbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" Sep 15 21:01:20.067887 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:20.067800 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" event={"ID":"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7","Type":"ContainerStarted","Data":"c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63"} Sep 15 21:01:20.082451 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:20.082426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" Sep 15 21:01:20.356281 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:20.356252 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6"] Sep 15 21:01:20.358144 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:01:20.358105 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970e2add_359e_4476_beab_786fc59cbd27.slice/crio-d707200f8878f64951fe9c5182acfd11aec8f0568b0d1a79e4b02414c4155f3e WatchSource:0}: Error finding container d707200f8878f64951fe9c5182acfd11aec8f0568b0d1a79e4b02414c4155f3e: Status 404 returned error can't find the container with id d707200f8878f64951fe9c5182acfd11aec8f0568b0d1a79e4b02414c4155f3e Sep 15 21:01:20.984331 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:20.984286 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:01:20.984795 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:20.984765 2579 patch_prober.go:28] interesting pod/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 15 21:01:20.984902 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:20.984833 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 15 21:01:21.075185 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:21.075019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" event={"ID":"970e2add-359e-4476-beab-786fc59cbd27","Type":"ContainerStarted","Data":"4436f955620e831db2ae525d46c818d74a18025f231507054cdd9349b6989340"} Sep 15 21:01:21.075185 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:21.075093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" event={"ID":"970e2add-359e-4476-beab-786fc59cbd27","Type":"ContainerStarted","Data":"d707200f8878f64951fe9c5182acfd11aec8f0568b0d1a79e4b02414c4155f3e"} Sep 15 21:01:21.096781 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:21.096705 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" podStartSLOduration=2.096680899 podStartE2EDuration="2.096680899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:01:21.095167262 +0000 UTC m=+1478.289120719" watchObservedRunningTime="2026-09-15 21:01:21.096680899 +0000 UTC m=+1478.290634356" Sep 15 21:01:21.984430 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:21.984389 2579 patch_prober.go:28] interesting pod/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 15 21:01:21.984765 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:21.984731 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 15 21:01:22.086881 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:22.086835 2579 generic.go:356] "Generic (PLEG): container finished" podID="970e2add-359e-4476-beab-786fc59cbd27" containerID="4436f955620e831db2ae525d46c818d74a18025f231507054cdd9349b6989340" exitCode=0 Sep 15 21:01:22.087386 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:22.086916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" event={"ID":"970e2add-359e-4476-beab-786fc59cbd27","Type":"ContainerDied","Data":"4436f955620e831db2ae525d46c818d74a18025f231507054cdd9349b6989340"} Sep 15 21:01:22.985153 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:22.985116 2579 patch_prober.go:28] interesting pod/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 15 21:01:22.985700 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:22.985670 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 15 21:01:23.257244 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:23.257220 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" Sep 15 21:01:23.338954 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:23.338922 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/970e2add-359e-4476-beab-786fc59cbd27-maas-api-service-ca\") pod \"970e2add-359e-4476-beab-786fc59cbd27\" (UID: \"970e2add-359e-4476-beab-786fc59cbd27\") " Sep 15 21:01:23.339256 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:23.339232 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970e2add-359e-4476-beab-786fc59cbd27-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "970e2add-359e-4476-beab-786fc59cbd27" (UID: "970e2add-359e-4476-beab-786fc59cbd27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:01:23.440002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:23.439962 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/970e2add-359e-4476-beab-786fc59cbd27-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:23.984963 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:23.984926 2579 patch_prober.go:28] interesting pod/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 15 21:01:23.985260 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:23.985231 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 15 21:01:24.104720 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:24.104689 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" Sep 15 21:01:24.104881 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:24.104716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6" event={"ID":"970e2add-359e-4476-beab-786fc59cbd27","Type":"ContainerDied","Data":"d707200f8878f64951fe9c5182acfd11aec8f0568b0d1a79e4b02414c4155f3e"} Sep 15 21:01:24.104881 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:24.104768 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d707200f8878f64951fe9c5182acfd11aec8f0568b0d1a79e4b02414c4155f3e" Sep 15 21:01:24.985151 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:24.985118 2579 patch_prober.go:28] interesting pod/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 15 21:01:24.985786 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:24.985751 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 15 21:01:25.457346 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:25.457311 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5"] Sep 15 21:01:25.988723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:25.988686 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:01:25.989181 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:25.988977 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:01:25.989807 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:25.989790 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:01:30.295088 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.295056 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr"] Sep 15 21:01:30.297854 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.297830 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-988a09-29825100-mk5tr"] Sep 15 21:01:30.320647 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.320611 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh"] Sep 15 21:01:30.320851 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.320810 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" podUID="2de27a99-0831-464e-b503-5927b83e8c2d" containerName="payload-processing" containerID="cri-o://671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950" gracePeriod=30 Sep 15 21:01:30.482046 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.482012 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6"] Sep 15 21:01:30.488769 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.488740 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-988a09-hpbm6"] Sep 15 21:01:30.570572 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.570547 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 21:01:30.617928 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.617891 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-584fz\" (UniqueName: \"kubernetes.io/projected/2de27a99-0831-464e-b503-5927b83e8c2d-kube-api-access-584fz\") pod \"2de27a99-0831-464e-b503-5927b83e8c2d\" (UID: \"2de27a99-0831-464e-b503-5927b83e8c2d\") " Sep 15 21:01:30.618116 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.617996 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de27a99-0831-464e-b503-5927b83e8c2d-plugins-config-volume\") pod \"2de27a99-0831-464e-b503-5927b83e8c2d\" (UID: \"2de27a99-0831-464e-b503-5927b83e8c2d\") " Sep 15 21:01:30.618346 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.618322 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de27a99-0831-464e-b503-5927b83e8c2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2de27a99-0831-464e-b503-5927b83e8c2d" (UID: "2de27a99-0831-464e-b503-5927b83e8c2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:01:30.618430 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.618408 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de27a99-0831-464e-b503-5927b83e8c2d-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:30.620181 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.620156 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de27a99-0831-464e-b503-5927b83e8c2d-kube-api-access-584fz" (OuterVolumeSpecName: "kube-api-access-584fz") pod "2de27a99-0831-464e-b503-5927b83e8c2d" (UID: "2de27a99-0831-464e-b503-5927b83e8c2d"). InnerVolumeSpecName "kube-api-access-584fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:01:30.719033 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:30.718998 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-584fz\" (UniqueName: \"kubernetes.io/projected/2de27a99-0831-464e-b503-5927b83e8c2d-kube-api-access-584fz\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:31.147985 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.147951 2579 generic.go:356] "Generic (PLEG): container finished" podID="2de27a99-0831-464e-b503-5927b83e8c2d" containerID="671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950" exitCode=0 Sep 15 21:01:31.148177 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.148012 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" Sep 15 21:01:31.148177 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.148036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" event={"ID":"2de27a99-0831-464e-b503-5927b83e8c2d","Type":"ContainerDied","Data":"671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950"} Sep 15 21:01:31.148177 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.148112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh" event={"ID":"2de27a99-0831-464e-b503-5927b83e8c2d","Type":"ContainerDied","Data":"990eea56810d861b923ba035e13eb1cf35f991f94573558b965c4261c3613174"} Sep 15 21:01:31.148177 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.148133 2579 scope.go:117] "RemoveContainer" containerID="671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950" Sep 15 21:01:31.158036 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.158017 2579 scope.go:117] "RemoveContainer" containerID="671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950" Sep 15 21:01:31.158295 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:01:31.158277 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950\": container with ID starting with 671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950 not found: ID does not exist" containerID="671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950" Sep 15 21:01:31.158372 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.158309 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950"} err="failed to get container status \"671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950\": rpc error: code = NotFound desc = could not find container \"671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950\": container with ID starting with 671b68bc962fae51b45982999897ad49f7405b5cc17d56b4a8a49bf13b360950 not found: ID does not exist" Sep 15 21:01:31.168846 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.168823 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh"] Sep 15 21:01:31.172777 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.172757 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-988a09-5f6876f445-7fdvh"] Sep 15 21:01:31.312547 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.312487 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2de27a99-0831-464e-b503-5927b83e8c2d" path="/var/lib/kubelet/pods/2de27a99-0831-464e-b503-5927b83e8c2d/volumes" Sep 15 21:01:31.312915 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.312866 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="970e2add-359e-4476-beab-786fc59cbd27" path="/var/lib/kubelet/pods/970e2add-359e-4476-beab-786fc59cbd27/volumes" Sep 15 21:01:31.313117 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:31.313104 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8b539d6-a1fe-45a5-9d58-1b71cde16b4a" path="/var/lib/kubelet/pods/c8b539d6-a1fe-45a5-9d58-1b71cde16b4a/volumes" Sep 15 21:01:40.503484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.503420 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" podUID="4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" containerName="main" containerID="cri-o://eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4" gracePeriod=45 Sep 15 21:01:40.752266 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.752244 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 21:01:40.828696 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.828668 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-home\") pod \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " Sep 15 21:01:40.828905 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.828715 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-model-cache\") pod \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " Sep 15 21:01:40.828905 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.828764 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kg54\" (UniqueName: \"kubernetes.io/projected/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-kube-api-access-7kg54\") pod \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " Sep 15 21:01:40.828905 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.828836 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tls-certs\") pod \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " Sep 15 21:01:40.828905 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.828882 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-dshm\") pod \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " Sep 15 21:01:40.829132 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.828922 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tmp-dir\") pod \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\" (UID: \"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a\") " Sep 15 21:01:40.829132 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.828917 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-home" (OuterVolumeSpecName: "home") pod "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" (UID: "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:40.829132 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.829037 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" (UID: "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:40.829302 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.829150 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" (UID: "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:40.829381 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.829367 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:40.829420 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.829382 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:40.829420 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.829392 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:40.831143 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.831116 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-kube-api-access-7kg54" (OuterVolumeSpecName: "kube-api-access-7kg54") pod "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" (UID: "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a"). InnerVolumeSpecName "kube-api-access-7kg54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:01:40.831535 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.831516 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-dshm" (OuterVolumeSpecName: "dshm") pod "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" (UID: "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:40.831618 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.831540 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" (UID: "4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:01:40.930225 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.930187 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kg54\" (UniqueName: \"kubernetes.io/projected/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-kube-api-access-7kg54\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:40.930225 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.930222 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:40.930225 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:40.930234 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:41.205248 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:41.205161 2579 generic.go:356] "Generic (PLEG): container finished" podID="4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" containerID="eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4" exitCode=0 Sep 15 21:01:41.205248 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:41.205201 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" Sep 15 21:01:41.205524 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:41.205248 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" event={"ID":"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a","Type":"ContainerDied","Data":"eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4"} Sep 15 21:01:41.205524 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:41.205304 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5" event={"ID":"4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a","Type":"ContainerDied","Data":"d5ce1197127b1cb895c6ad31484e6dede1e229d5f997e37e6dde4aad843e1418"} Sep 15 21:01:41.205524 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:41.205328 2579 scope.go:117] "RemoveContainer" containerID="eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4" Sep 15 21:01:41.215425 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:41.215408 2579 scope.go:117] "RemoveContainer" containerID="eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4" Sep 15 21:01:41.215709 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:01:41.215688 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4\": container with ID starting with eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4 not found: ID does not exist" containerID="eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4" Sep 15 21:01:41.215789 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:41.215721 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4"} err="failed to get container status \"eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4\": rpc error: code = NotFound desc = could not find container \"eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4\": container with ID starting with eda1ba293f11e7832aa8e421212f9c1e3e3a72c225468c2063a26d72b458aae4 not found: ID does not exist" Sep 15 21:01:41.229530 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:41.226415 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5"] Sep 15 21:01:41.233376 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:41.233335 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-988a09/test-model-988a09-kserve-86bf4dffbc-pgzr5"] Sep 15 21:01:41.312079 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:41.312043 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" path="/var/lib/kubelet/pods/4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a/volumes" Sep 15 21:01:42.322065 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:42.322033 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c"] Sep 15 21:01:42.322439 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:42.322213 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" containerID="cri-o://f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625" gracePeriod=30 Sep 15 21:01:43.466620 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:43.466588 2579 scope.go:117] "RemoveContainer" containerID="10445952fd22dfcbc9a8522f2a76224d80996a4fc64432f3bf4c664b21ff4e9d" Sep 15 21:01:47.569930 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.569901 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 21:01:47.697703 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.697615 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-credential-socket\") pod \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " Sep 15 21:01:47.697703 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.697661 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-certs\") pod \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " Sep 15 21:01:47.697703 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.697678 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-envoy\") pod \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " Sep 15 21:01:47.697703 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.697702 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-socket\") pod \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " Sep 15 21:01:47.698038 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.697723 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-data\") pod \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " Sep 15 21:01:47.698038 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.697748 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6gsz\" (UniqueName: \"kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-kube-api-access-t6gsz\") pod \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " Sep 15 21:01:47.698038 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.697844 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istiod-ca-cert\") pod \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " Sep 15 21:01:47.698038 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.697940 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" (UID: "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:47.698038 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.697957 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" (UID: "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:47.698244 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.698035 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" (UID: "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:47.698244 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.698101 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" (UID: "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:47.698244 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.698167 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-token\") pod \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " Sep 15 21:01:47.698244 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.698190 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" (UID: "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:01:47.698244 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.698242 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-podinfo\") pod \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\" (UID: \"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877\") " Sep 15 21:01:47.698640 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.698624 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:47.698814 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.698642 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:47.698814 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.698651 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:47.698814 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.698660 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:47.698814 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.698668 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:47.700061 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.700030 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-kube-api-access-t6gsz" (OuterVolumeSpecName: "kube-api-access-t6gsz") pod "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" (UID: "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877"). InnerVolumeSpecName "kube-api-access-t6gsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:01:47.700167 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.700083 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" (UID: "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:47.700294 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.700280 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" (UID: "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:01:47.700412 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.700399 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" (UID: "c0a1a8b0-2bc8-40f8-8279-94c28f4a9877"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:01:47.799823 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.799790 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:47.799823 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.799819 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:47.799823 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.799830 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:47.800054 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:47.799840 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6gsz\" (UniqueName: \"kubernetes.io/projected/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877-kube-api-access-t6gsz\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:01:48.247157 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.247120 2579 generic.go:356] "Generic (PLEG): container finished" podID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerID="f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625" exitCode=0 Sep 15 21:01:48.247330 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.247170 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" Sep 15 21:01:48.247330 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.247194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" event={"ID":"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877","Type":"ContainerDied","Data":"f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625"} Sep 15 21:01:48.247330 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.247252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c" event={"ID":"c0a1a8b0-2bc8-40f8-8279-94c28f4a9877","Type":"ContainerDied","Data":"8096a2e68608689b5bc3f0dfdab67faebe5f207cb33970349e8eba71e793e452"} Sep 15 21:01:48.247330 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.247274 2579 scope.go:117] "RemoveContainer" containerID="f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625" Sep 15 21:01:48.257005 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.256974 2579 scope.go:117] "RemoveContainer" containerID="825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97" Sep 15 21:01:48.265588 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.265569 2579 scope.go:117] "RemoveContainer" containerID="f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625" Sep 15 21:01:48.265870 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:01:48.265850 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625\": container with ID starting with f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625 not found: ID does not exist" containerID="f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625" Sep 15 21:01:48.265948 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.265878 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625"} err="failed to get container status \"f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625\": rpc error: code = NotFound desc = could not find container \"f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625\": container with ID starting with f2d87cfcaeaae5a50aa314aed2d21b40841725276041f1b484286467c0c37625 not found: ID does not exist" Sep 15 21:01:48.265948 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.265896 2579 scope.go:117] "RemoveContainer" containerID="825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97" Sep 15 21:01:48.266142 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:01:48.266127 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97\": container with ID starting with 825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97 not found: ID does not exist" containerID="825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97" Sep 15 21:01:48.266184 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.266147 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97"} err="failed to get container status \"825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97\": rpc error: code = NotFound desc = could not find container \"825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97\": container with ID starting with 825d5557a6cc14b45107be56fd3758f1582e65f612359e9c3f4a0a3ddf533f97 not found: ID does not exist" Sep 15 21:01:48.269752 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.269729 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c"] Sep 15 21:01:48.275409 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:48.275389 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-988a09-openshift-default-65bd6db9b6-8zb2c"] Sep 15 21:01:49.311143 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:49.311098 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" path="/var/lib/kubelet/pods/c0a1a8b0-2bc8-40f8-8279-94c28f4a9877/volumes" Sep 15 21:01:50.813957 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.813923 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z"] Sep 15 21:01:50.814443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814326 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" containerName="main" Sep 15 21:01:50.814443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814338 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" containerName="main" Sep 15 21:01:50.814443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814347 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2de27a99-0831-464e-b503-5927b83e8c2d" containerName="payload-processing" Sep 15 21:01:50.814443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814352 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2de27a99-0831-464e-b503-5927b83e8c2d" containerName="payload-processing" Sep 15 21:01:50.814443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814364 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970e2add-359e-4476-beab-786fc59cbd27" containerName="revoke-keys" Sep 15 21:01:50.814443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814370 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="970e2add-359e-4476-beab-786fc59cbd27" containerName="revoke-keys" Sep 15 21:01:50.814443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814378 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" Sep 15 21:01:50.814443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814383 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" Sep 15 21:01:50.814443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814402 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" Sep 15 21:01:50.814443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814409 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" Sep 15 21:01:50.814833 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814477 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="970e2add-359e-4476-beab-786fc59cbd27" containerName="revoke-keys" Sep 15 21:01:50.814833 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814492 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" Sep 15 21:01:50.814833 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814511 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a1a8b0-2bc8-40f8-8279-94c28f4a9877" containerName="istio-proxy" Sep 15 21:01:50.814833 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814518 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2de27a99-0831-464e-b503-5927b83e8c2d" containerName="payload-processing" Sep 15 21:01:50.814833 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.814523 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0e80e5-72ec-4c12-a2bc-1f9e01d8803a" containerName="main" Sep 15 21:01:50.818679 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.818663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:01:50.821035 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.821012 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-280fb4\"" Sep 15 21:01:50.821174 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.821092 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-280fb4-dockercfg-z2lvb\"" Sep 15 21:01:50.825806 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.825778 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z"] Sep 15 21:01:50.864037 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.864004 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5"] Sep 15 21:01:50.867533 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.867514 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:01:50.879264 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.879240 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5"] Sep 15 21:01:50.929963 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.929918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73877a8-533a-4d00-9755-917bee4519d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z\" (UID: \"c73877a8-533a-4d00-9755-917bee4519d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:01:50.930147 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:50.930005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fftwg\" (UniqueName: \"kubernetes.io/projected/c73877a8-533a-4d00-9755-917bee4519d4-kube-api-access-fftwg\") pod \"payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z\" (UID: \"c73877a8-533a-4d00-9755-917bee4519d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:01:51.030899 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.030865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246fa19f-ba51-465a-818c-4c19d3786857-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5\" (UID: \"246fa19f-ba51-465a-818c-4c19d3786857\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:01:51.030899 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.030903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqjb\" (UniqueName: \"kubernetes.io/projected/246fa19f-ba51-465a-818c-4c19d3786857-kube-api-access-dcqjb\") pod \"payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5\" (UID: \"246fa19f-ba51-465a-818c-4c19d3786857\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:01:51.031107 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.030928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73877a8-533a-4d00-9755-917bee4519d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z\" (UID: \"c73877a8-533a-4d00-9755-917bee4519d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:01:51.031107 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.031061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fftwg\" (UniqueName: \"kubernetes.io/projected/c73877a8-533a-4d00-9755-917bee4519d4-kube-api-access-fftwg\") pod \"payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z\" (UID: \"c73877a8-533a-4d00-9755-917bee4519d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:01:51.031640 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.031623 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73877a8-533a-4d00-9755-917bee4519d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z\" (UID: \"c73877a8-533a-4d00-9755-917bee4519d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:01:51.039441 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.039411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fftwg\" (UniqueName: \"kubernetes.io/projected/c73877a8-533a-4d00-9755-917bee4519d4-kube-api-access-fftwg\") pod \"payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z\" (UID: \"c73877a8-533a-4d00-9755-917bee4519d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:01:51.130005 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.129919 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:01:51.132393 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.132370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246fa19f-ba51-465a-818c-4c19d3786857-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5\" (UID: \"246fa19f-ba51-465a-818c-4c19d3786857\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:01:51.132464 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.132410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqjb\" (UniqueName: \"kubernetes.io/projected/246fa19f-ba51-465a-818c-4c19d3786857-kube-api-access-dcqjb\") pod \"payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5\" (UID: \"246fa19f-ba51-465a-818c-4c19d3786857\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:01:51.133382 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.133061 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246fa19f-ba51-465a-818c-4c19d3786857-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5\" (UID: \"246fa19f-ba51-465a-818c-4c19d3786857\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:01:51.141193 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.141161 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqjb\" (UniqueName: \"kubernetes.io/projected/246fa19f-ba51-465a-818c-4c19d3786857-kube-api-access-dcqjb\") pod \"payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5\" (UID: \"246fa19f-ba51-465a-818c-4c19d3786857\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:01:51.178978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.178535 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:01:51.400285 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.400263 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z"] Sep 15 21:01:51.402521 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:01:51.402480 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73877a8_533a_4d00_9755_917bee4519d4.slice/crio-fcd5fc0aa663cf72531d5e2d8f411e9ac3202d8e8413dcd3c7408e65d1c79eab WatchSource:0}: Error finding container fcd5fc0aa663cf72531d5e2d8f411e9ac3202d8e8413dcd3c7408e65d1c79eab: Status 404 returned error can't find the container with id fcd5fc0aa663cf72531d5e2d8f411e9ac3202d8e8413dcd3c7408e65d1c79eab Sep 15 21:01:51.464815 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:51.464789 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5"] Sep 15 21:01:51.468225 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:01:51.468202 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246fa19f_ba51_465a_818c_4c19d3786857.slice/crio-d7c39fbbb007075065d03e524ba6f33d944b548b5e02a038ef54a1ea4147d63b WatchSource:0}: Error finding container d7c39fbbb007075065d03e524ba6f33d944b548b5e02a038ef54a1ea4147d63b: Status 404 returned error can't find the container with id d7c39fbbb007075065d03e524ba6f33d944b548b5e02a038ef54a1ea4147d63b Sep 15 21:01:52.274333 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:52.274284 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" event={"ID":"246fa19f-ba51-465a-818c-4c19d3786857","Type":"ContainerStarted","Data":"ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c"} Sep 15 21:01:52.274824 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:52.274362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" event={"ID":"246fa19f-ba51-465a-818c-4c19d3786857","Type":"ContainerStarted","Data":"d7c39fbbb007075065d03e524ba6f33d944b548b5e02a038ef54a1ea4147d63b"} Sep 15 21:01:52.274824 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:52.274394 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:01:52.275973 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:52.275946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" event={"ID":"c73877a8-533a-4d00-9755-917bee4519d4","Type":"ContainerStarted","Data":"c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0"} Sep 15 21:01:52.276091 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:52.275986 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" event={"ID":"c73877a8-533a-4d00-9755-917bee4519d4","Type":"ContainerStarted","Data":"fcd5fc0aa663cf72531d5e2d8f411e9ac3202d8e8413dcd3c7408e65d1c79eab"} Sep 15 21:01:52.276091 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:52.276024 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:01:52.291201 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:52.291158 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" podStartSLOduration=2.291143651 podStartE2EDuration="2.291143651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:01:52.289254482 +0000 UTC m=+1509.483208116" watchObservedRunningTime="2026-09-15 21:01:52.291143651 +0000 UTC m=+1509.485097105" Sep 15 21:01:52.304642 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:01:52.304591 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" podStartSLOduration=2.304572331 podStartE2EDuration="2.304572331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:01:52.302801888 +0000 UTC m=+1509.496755343" watchObservedRunningTime="2026-09-15 21:01:52.304572331 +0000 UTC m=+1509.498525789" Sep 15 21:02:03.284306 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:03.284272 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:02:03.284771 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:03.284357 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:02:04.069726 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.069691 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp"] Sep 15 21:02:04.073542 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.073526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.075555 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.075526 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-65001159\"/\"default-dockercfg-qd7zp\"" Sep 15 21:02:04.075681 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.075560 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-65001159\"/\"kube-root-ca.crt\"" Sep 15 21:02:04.076337 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.076319 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-65001159\"/\"e2e-lifecycle-model-65001159-kserve-self-signed-certs\"" Sep 15 21:02:04.076441 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.076360 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-65001159\"/\"openshift-service-ca.crt\"" Sep 15 21:02:04.081584 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.081562 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp"] Sep 15 21:02:04.148772 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.148742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-tmp-dir\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.148772 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.148771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73f36f1-fc73-41e3-9143-38d769f632f1-tls-certs\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.149001 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.148803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-model-cache\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.149001 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.148898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-home\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.149001 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.148929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-dshm\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.149172 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.149002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2fqn\" (UniqueName: \"kubernetes.io/projected/b73f36f1-fc73-41e3-9143-38d769f632f1-kube-api-access-t2fqn\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.250201 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.250160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-home\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.250201 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.250207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-dshm\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.250449 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.250234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fqn\" (UniqueName: \"kubernetes.io/projected/b73f36f1-fc73-41e3-9143-38d769f632f1-kube-api-access-t2fqn\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.250449 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.250309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-tmp-dir\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.250449 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.250337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73f36f1-fc73-41e3-9143-38d769f632f1-tls-certs\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.250449 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.250367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-model-cache\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.250712 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.250688 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-home\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.250768 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.250717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-tmp-dir\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.250810 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.250792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-model-cache\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.252594 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.252572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-dshm\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.252943 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.252923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73f36f1-fc73-41e3-9143-38d769f632f1-tls-certs\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.257285 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.257256 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2fqn\" (UniqueName: \"kubernetes.io/projected/b73f36f1-fc73-41e3-9143-38d769f632f1-kube-api-access-t2fqn\") pod \"e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.385301 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.385173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:04.662094 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:04.662069 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp"] Sep 15 21:02:04.664121 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:02:04.664097 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb73f36f1_fc73_41e3_9143_38d769f632f1.slice/crio-d66db213f7acae8076a3295f4b1100d14cec3dbdb6adbd8c3685d21186e8bbae WatchSource:0}: Error finding container d66db213f7acae8076a3295f4b1100d14cec3dbdb6adbd8c3685d21186e8bbae: Status 404 returned error can't find the container with id d66db213f7acae8076a3295f4b1100d14cec3dbdb6adbd8c3685d21186e8bbae Sep 15 21:02:05.350535 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:05.350485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" event={"ID":"b73f36f1-fc73-41e3-9143-38d769f632f1","Type":"ContainerStarted","Data":"cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6"} Sep 15 21:02:05.350699 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:05.350546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" event={"ID":"b73f36f1-fc73-41e3-9143-38d769f632f1","Type":"ContainerStarted","Data":"d66db213f7acae8076a3295f4b1100d14cec3dbdb6adbd8c3685d21186e8bbae"} Sep 15 21:02:05.365901 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:05.365857 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" podStartSLOduration=1.36584109 podStartE2EDuration="1.36584109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:02:05.364781691 +0000 UTC m=+1522.558735145" watchObservedRunningTime="2026-09-15 21:02:05.36584109 +0000 UTC m=+1522.559794581" Sep 15 21:02:14.386022 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:14.385983 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:14.386022 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:14.386030 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:14.399475 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:14.399447 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:14.414450 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:14.414420 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:02:17.332052 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.332016 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl"] Sep 15 21:02:17.336118 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.336098 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.339629 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.339604 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-728877-openshift-default-dockercfg-2pf7h\"" Sep 15 21:02:17.347258 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.347231 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl"] Sep 15 21:02:17.488939 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.488894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-token\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.488939 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.488946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-credential-socket\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.489197 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.488976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-certs\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.489197 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.489016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-data\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.489197 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.489071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-podinfo\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.489197 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.489099 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xhd\" (UniqueName: \"kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-kube-api-access-k2xhd\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.489365 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.489196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-socket\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.489365 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.489246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-envoy\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.489365 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.489313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52fe772-acba-4dbe-9a12-543c7c3b0955-istiod-ca-cert\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.590397 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-socket\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.590397 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-envoy\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.590679 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52fe772-acba-4dbe-9a12-543c7c3b0955-istiod-ca-cert\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.590679 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-token\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.590679 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-credential-socket\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.590679 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-certs\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.590679 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-data\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.590679 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-podinfo\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.591156 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xhd\" (UniqueName: \"kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-kube-api-access-k2xhd\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.591156 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-socket\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.591156 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.590933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-credential-socket\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.591156 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.591090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-data\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.591335 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.591180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-certs\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.591335 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.591188 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52fe772-acba-4dbe-9a12-543c7c3b0955-istiod-ca-cert\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.593722 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.593694 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-envoy\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.593861 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.593802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-podinfo\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.598707 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.598682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-token\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.598899 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.598876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xhd\" (UniqueName: \"kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-kube-api-access-k2xhd\") pod \"e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.648525 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.648462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:17.929627 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.929580 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl"] Sep 15 21:02:17.932043 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:02:17.932010 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52fe772_acba_4dbe_9a12_543c7c3b0955.slice/crio-30baf4d301e76b7ca470bac907b4041aa098f0af0c7cbb0cab4fc44966e3622d WatchSource:0}: Error finding container 30baf4d301e76b7ca470bac907b4041aa098f0af0c7cbb0cab4fc44966e3622d: Status 404 returned error can't find the container with id 30baf4d301e76b7ca470bac907b4041aa098f0af0c7cbb0cab4fc44966e3622d Sep 15 21:02:17.934327 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.934292 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:02:17.934422 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.934374 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:02:17.934485 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:17.934417 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:02:18.427145 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:18.427103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" event={"ID":"c52fe772-acba-4dbe-9a12-543c7c3b0955","Type":"ContainerStarted","Data":"d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db"} Sep 15 21:02:18.427610 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:18.427165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" event={"ID":"c52fe772-acba-4dbe-9a12-543c7c3b0955","Type":"ContainerStarted","Data":"30baf4d301e76b7ca470bac907b4041aa098f0af0c7cbb0cab4fc44966e3622d"} Sep 15 21:02:18.447331 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:18.447280 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" podStartSLOduration=1.4472622880000001 podStartE2EDuration="1.447262288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:02:18.445601207 +0000 UTC m=+1535.639554662" watchObservedRunningTime="2026-09-15 21:02:18.447262288 +0000 UTC m=+1535.641216140" Sep 15 21:02:18.649592 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:18.649554 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:18.655016 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:18.654991 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:19.433579 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:19.433549 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:19.434567 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:19.434546 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:43.506654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:43.506623 2579 scope.go:117] "RemoveContainer" containerID="d0b0547aae8ec8cca5ace148ed9a14e33eb6748674a2e6e264216384f7592292" Sep 15 21:02:43.515986 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:43.515961 2579 scope.go:117] "RemoveContainer" containerID="0e24d0aefd446fba5d86d2be52602524e7802e32b0b32c6f5f1a8823d1f9dbe5" Sep 15 21:02:43.528236 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:43.528215 2579 scope.go:117] "RemoveContainer" containerID="ffc751baeb6917f73283e8b19c068baeb070ed39bc4c40725846ef22aadd29a8" Sep 15 21:02:46.588110 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:46.588020 2579 generic.go:356] "Generic (PLEG): container finished" podID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerID="d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db" exitCode=0 Sep 15 21:02:46.588651 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:46.588103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" event={"ID":"c52fe772-acba-4dbe-9a12-543c7c3b0955","Type":"ContainerDied","Data":"d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db"} Sep 15 21:02:46.588724 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:46.588676 2579 scope.go:117] "RemoveContainer" containerID="d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db" Sep 15 21:02:46.589929 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:46.589901 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:02:46.590039 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:46.589983 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:02:46.590039 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:46.590026 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:02:47.595778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:47.595736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" event={"ID":"c52fe772-acba-4dbe-9a12-543c7c3b0955","Type":"ContainerStarted","Data":"70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b"} Sep 15 21:02:47.648885 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:47.648838 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:47.649782 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:47.649755 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:02:47.649921 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:47.649805 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 15 21:02:48.649182 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:48.649149 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:02:48.649606 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:48.649204 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 15 21:02:49.434103 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:49.434066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:49.649079 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:49.649044 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:02:49.649268 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:49.649099 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 15 21:02:50.649455 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:50.649419 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:02:50.649859 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:50.649478 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 15 21:02:51.649239 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:51.649202 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:02:51.649420 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:51.649253 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 15 21:02:52.649783 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:52.649749 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:02:52.650168 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:52.649807 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 15 21:02:53.653730 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:53.653698 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:02:53.654749 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:02:53.654727 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:03:34.167367 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.167331 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp"] Sep 15 21:03:34.171359 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.171334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.173625 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.173602 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"kube-root-ca.crt\"" Sep 15 21:03:34.174614 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.174592 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"default-dockercfg-m9ttf\"" Sep 15 21:03:34.174719 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.174625 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"auth-test-model-280fb4-kserve-self-signed-certs\"" Sep 15 21:03:34.174719 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.174598 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"openshift-service-ca.crt\"" Sep 15 21:03:34.180265 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.180243 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp"] Sep 15 21:03:34.271619 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.271587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-dshm\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.271619 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.271621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cx6n\" (UniqueName: \"kubernetes.io/projected/5bfd3d03-d232-4d83-a394-7e75e3db412d-kube-api-access-5cx6n\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.271878 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.271669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-tmp-dir\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.271878 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.271703 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfd3d03-d232-4d83-a394-7e75e3db412d-tls-certs\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.271878 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.271725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-model-cache\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.272032 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.271936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-home\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.373384 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.373292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-home\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.373384 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.373375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-dshm\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.373667 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.373429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cx6n\" (UniqueName: \"kubernetes.io/projected/5bfd3d03-d232-4d83-a394-7e75e3db412d-kube-api-access-5cx6n\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.373667 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.373483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-tmp-dir\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.373667 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.373541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfd3d03-d232-4d83-a394-7e75e3db412d-tls-certs\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.373667 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.373565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-model-cache\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.373881 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.373749 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-home\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.373881 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.373835 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-tmp-dir\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.374011 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.373988 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-model-cache\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.375745 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.375717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-dshm\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.376087 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.376070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfd3d03-d232-4d83-a394-7e75e3db412d-tls-certs\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.382314 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.382283 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cx6n\" (UniqueName: \"kubernetes.io/projected/5bfd3d03-d232-4d83-a394-7e75e3db412d-kube-api-access-5cx6n\") pod \"auth-test-model-280fb4-kserve-77df8dc6c-45ccp\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.482039 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.481949 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:34.753547 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.753520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp"] Sep 15 21:03:34.756084 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:03:34.756056 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bfd3d03_d232_4d83_a394_7e75e3db412d.slice/crio-872391a3a588fd3d49f50c73a3f3e73496579c124ca70af9a127fa1b845393b1 WatchSource:0}: Error finding container 872391a3a588fd3d49f50c73a3f3e73496579c124ca70af9a127fa1b845393b1: Status 404 returned error can't find the container with id 872391a3a588fd3d49f50c73a3f3e73496579c124ca70af9a127fa1b845393b1 Sep 15 21:03:34.863966 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.863935 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" event={"ID":"5bfd3d03-d232-4d83-a394-7e75e3db412d","Type":"ContainerStarted","Data":"52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c"} Sep 15 21:03:34.864120 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.863979 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" event={"ID":"5bfd3d03-d232-4d83-a394-7e75e3db412d","Type":"ContainerStarted","Data":"872391a3a588fd3d49f50c73a3f3e73496579c124ca70af9a127fa1b845393b1"} Sep 15 21:03:34.883512 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:34.883439 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" podStartSLOduration=0.883419944 podStartE2EDuration="883.419944ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:03:34.87953798 +0000 UTC m=+1612.073491439" watchObservedRunningTime="2026-09-15 21:03:34.883419944 +0000 UTC m=+1612.077373397" Sep 15 21:03:44.482403 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:44.482363 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:44.482874 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:44.482439 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:44.495158 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:44.495135 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:44.929367 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:44.929335 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:03:52.764744 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:52.764704 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr"] Sep 15 21:03:52.768745 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:52.768721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" Sep 15 21:03:52.771197 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:52.771176 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:03:52.776941 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:52.776697 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr"] Sep 15 21:03:52.861908 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:52.861862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1f0d88-e092-45e5-ab39-ced258c11b50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65001159-m2qmr\" (UID: \"4a1f0d88-e092-45e5-ab39-ced258c11b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" Sep 15 21:03:52.962778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:52.962747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1f0d88-e092-45e5-ab39-ced258c11b50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65001159-m2qmr\" (UID: \"4a1f0d88-e092-45e5-ab39-ced258c11b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" Sep 15 21:03:52.963578 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:52.963558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1f0d88-e092-45e5-ab39-ced258c11b50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65001159-m2qmr\" (UID: \"4a1f0d88-e092-45e5-ab39-ced258c11b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" Sep 15 21:03:53.081168 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:53.081075 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" Sep 15 21:03:53.353182 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:53.353159 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr"] Sep 15 21:03:53.355120 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:03:53.355095 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1f0d88_e092_45e5_ab39_ced258c11b50.slice/crio-d0c8e26fe4dc45b82afd1e1913783598535dd62539793fabdc8e061715cd28ed WatchSource:0}: Error finding container d0c8e26fe4dc45b82afd1e1913783598535dd62539793fabdc8e061715cd28ed: Status 404 returned error can't find the container with id d0c8e26fe4dc45b82afd1e1913783598535dd62539793fabdc8e061715cd28ed Sep 15 21:03:53.972613 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:53.972509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" event={"ID":"4a1f0d88-e092-45e5-ab39-ced258c11b50","Type":"ContainerStarted","Data":"f5f45b168bbe5758d0e06f520933f72bc24c1e5878bed9b9ba7666a13b674861"} Sep 15 21:03:53.972613 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:53.972574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" event={"ID":"4a1f0d88-e092-45e5-ab39-ced258c11b50","Type":"ContainerStarted","Data":"d0c8e26fe4dc45b82afd1e1913783598535dd62539793fabdc8e061715cd28ed"} Sep 15 21:03:53.988619 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:53.988575 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" podStartSLOduration=1.9885578929999999 podStartE2EDuration="1.988557893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:03:53.987393648 +0000 UTC m=+1631.181347104" watchObservedRunningTime="2026-09-15 21:03:53.988557893 +0000 UTC m=+1631.182511347" Sep 15 21:03:54.979756 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:54.979715 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a1f0d88-e092-45e5-ab39-ced258c11b50" containerID="f5f45b168bbe5758d0e06f520933f72bc24c1e5878bed9b9ba7666a13b674861" exitCode=0 Sep 15 21:03:54.980170 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:54.979793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" event={"ID":"4a1f0d88-e092-45e5-ab39-ced258c11b50","Type":"ContainerDied","Data":"f5f45b168bbe5758d0e06f520933f72bc24c1e5878bed9b9ba7666a13b674861"} Sep 15 21:03:56.120619 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:56.120593 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" Sep 15 21:03:56.195068 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:56.195034 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1f0d88-e092-45e5-ab39-ced258c11b50-maas-api-service-ca\") pod \"4a1f0d88-e092-45e5-ab39-ced258c11b50\" (UID: \"4a1f0d88-e092-45e5-ab39-ced258c11b50\") " Sep 15 21:03:56.195364 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:56.195341 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1f0d88-e092-45e5-ab39-ced258c11b50-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a1f0d88-e092-45e5-ab39-ced258c11b50" (UID: "4a1f0d88-e092-45e5-ab39-ced258c11b50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:03:56.296850 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:56.296817 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1f0d88-e092-45e5-ab39-ced258c11b50-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:03:56.991098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:56.991066 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" Sep 15 21:03:56.991277 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:56.991107 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr" event={"ID":"4a1f0d88-e092-45e5-ab39-ced258c11b50","Type":"ContainerDied","Data":"d0c8e26fe4dc45b82afd1e1913783598535dd62539793fabdc8e061715cd28ed"} Sep 15 21:03:56.991277 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:03:56.991148 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0c8e26fe4dc45b82afd1e1913783598535dd62539793fabdc8e061715cd28ed" Sep 15 21:04:08.289230 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.289197 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp"] Sep 15 21:04:08.289846 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.289537 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" podUID="86e6587a-a3c8-4352-8871-7ae10c1e7e58" containerName="payload-processing" containerID="cri-o://b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d" gracePeriod=30 Sep 15 21:04:08.502911 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.502876 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr"] Sep 15 21:04:08.504972 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.504949 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65001159-m2qmr"] Sep 15 21:04:08.536170 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.536141 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:04:08.622484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.622412 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86e6587a-a3c8-4352-8871-7ae10c1e7e58-plugins-config-volume\") pod \"86e6587a-a3c8-4352-8871-7ae10c1e7e58\" (UID: \"86e6587a-a3c8-4352-8871-7ae10c1e7e58\") " Sep 15 21:04:08.622664 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.622543 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qgrv\" (UniqueName: \"kubernetes.io/projected/86e6587a-a3c8-4352-8871-7ae10c1e7e58-kube-api-access-8qgrv\") pod \"86e6587a-a3c8-4352-8871-7ae10c1e7e58\" (UID: \"86e6587a-a3c8-4352-8871-7ae10c1e7e58\") " Sep 15 21:04:08.622775 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.622751 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e6587a-a3c8-4352-8871-7ae10c1e7e58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86e6587a-a3c8-4352-8871-7ae10c1e7e58" (UID: "86e6587a-a3c8-4352-8871-7ae10c1e7e58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:04:08.624725 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.624702 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e6587a-a3c8-4352-8871-7ae10c1e7e58-kube-api-access-8qgrv" (OuterVolumeSpecName: "kube-api-access-8qgrv") pod "86e6587a-a3c8-4352-8871-7ae10c1e7e58" (UID: "86e6587a-a3c8-4352-8871-7ae10c1e7e58"). InnerVolumeSpecName "kube-api-access-8qgrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:04:08.723335 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.723299 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qgrv\" (UniqueName: \"kubernetes.io/projected/86e6587a-a3c8-4352-8871-7ae10c1e7e58-kube-api-access-8qgrv\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:08.723335 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:08.723331 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86e6587a-a3c8-4352-8871-7ae10c1e7e58-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:09.058701 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.058660 2579 generic.go:356] "Generic (PLEG): container finished" podID="86e6587a-a3c8-4352-8871-7ae10c1e7e58" containerID="b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d" exitCode=0 Sep 15 21:04:09.058903 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.058705 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" Sep 15 21:04:09.058903 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.058725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" event={"ID":"86e6587a-a3c8-4352-8871-7ae10c1e7e58","Type":"ContainerDied","Data":"b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d"} Sep 15 21:04:09.058903 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.058774 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp" event={"ID":"86e6587a-a3c8-4352-8871-7ae10c1e7e58","Type":"ContainerDied","Data":"9d7a8c7eec7a372a3a165ae8f5501ccd1e05344357606967dff6454bcdd2c90b"} Sep 15 21:04:09.058903 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.058799 2579 scope.go:117] "RemoveContainer" containerID="b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d" Sep 15 21:04:09.068239 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.068219 2579 scope.go:117] "RemoveContainer" containerID="b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d" Sep 15 21:04:09.068512 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:04:09.068481 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d\": container with ID starting with b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d not found: ID does not exist" containerID="b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d" Sep 15 21:04:09.068564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.068523 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d"} err="failed to get container status \"b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d\": rpc error: code = NotFound desc = could not find container \"b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d\": container with ID starting with b2b4cc649794ed7cb8a8a8868920dfedae5d717dfcc17e88ba90ebb15c44a32d not found: ID does not exist" Sep 15 21:04:09.079233 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.079211 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp"] Sep 15 21:04:09.082426 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.082408 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65001159-dc75856b9-nq6hp"] Sep 15 21:04:09.313054 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.312973 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a1f0d88-e092-45e5-ab39-ced258c11b50" path="/var/lib/kubelet/pods/4a1f0d88-e092-45e5-ab39-ced258c11b50/volumes" Sep 15 21:04:09.313426 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:09.313369 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86e6587a-a3c8-4352-8871-7ae10c1e7e58" path="/var/lib/kubelet/pods/86e6587a-a3c8-4352-8871-7ae10c1e7e58/volumes" Sep 15 21:04:16.170266 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:16.164672 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52"] Sep 15 21:04:17.148263 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:17.148231 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp"] Sep 15 21:04:31.189325 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.189281 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" podUID="d07773c9-4cb0-4541-aaca-e1ddb32e915a" containerName="main" containerID="cri-o://0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153" gracePeriod=45 Sep 15 21:04:31.443061 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.443000 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 21:04:31.550171 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550137 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-home\") pod \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " Sep 15 21:04:31.550354 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550222 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-dshm\") pod \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " Sep 15 21:04:31.550354 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550278 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tls-certs\") pod \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " Sep 15 21:04:31.550354 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550305 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tmp-dir\") pod \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " Sep 15 21:04:31.550354 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550341 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzcrf\" (UniqueName: \"kubernetes.io/projected/d07773c9-4cb0-4541-aaca-e1ddb32e915a-kube-api-access-tzcrf\") pod \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " Sep 15 21:04:31.550613 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550394 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-model-cache\") pod \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\" (UID: \"d07773c9-4cb0-4541-aaca-e1ddb32e915a\") " Sep 15 21:04:31.550613 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550442 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-home" (OuterVolumeSpecName: "home") pod "d07773c9-4cb0-4541-aaca-e1ddb32e915a" (UID: "d07773c9-4cb0-4541-aaca-e1ddb32e915a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:04:31.550723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550630 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d07773c9-4cb0-4541-aaca-e1ddb32e915a" (UID: "d07773c9-4cb0-4541-aaca-e1ddb32e915a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:04:31.550723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550705 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d07773c9-4cb0-4541-aaca-e1ddb32e915a" (UID: "d07773c9-4cb0-4541-aaca-e1ddb32e915a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:04:31.550885 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550860 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:31.550946 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550886 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:31.550946 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.550898 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:31.552676 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.552644 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-dshm" (OuterVolumeSpecName: "dshm") pod "d07773c9-4cb0-4541-aaca-e1ddb32e915a" (UID: "d07773c9-4cb0-4541-aaca-e1ddb32e915a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:04:31.552676 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.552648 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d07773c9-4cb0-4541-aaca-e1ddb32e915a" (UID: "d07773c9-4cb0-4541-aaca-e1ddb32e915a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:04:31.552951 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.552933 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07773c9-4cb0-4541-aaca-e1ddb32e915a-kube-api-access-tzcrf" (OuterVolumeSpecName: "kube-api-access-tzcrf") pod "d07773c9-4cb0-4541-aaca-e1ddb32e915a" (UID: "d07773c9-4cb0-4541-aaca-e1ddb32e915a"). InnerVolumeSpecName "kube-api-access-tzcrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:04:31.652346 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.652298 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07773c9-4cb0-4541-aaca-e1ddb32e915a-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:31.652346 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.652341 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07773c9-4cb0-4541-aaca-e1ddb32e915a-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:31.652346 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:31.652353 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzcrf\" (UniqueName: \"kubernetes.io/projected/d07773c9-4cb0-4541-aaca-e1ddb32e915a-kube-api-access-tzcrf\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:32.168002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.167951 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" podUID="b73f36f1-fc73-41e3-9143-38d769f632f1" containerName="main" containerID="cri-o://cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6" gracePeriod=45 Sep 15 21:04:32.198646 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.198616 2579 generic.go:356] "Generic (PLEG): container finished" podID="d07773c9-4cb0-4541-aaca-e1ddb32e915a" containerID="0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153" exitCode=0 Sep 15 21:04:32.199042 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.198677 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" Sep 15 21:04:32.199042 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.198685 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" event={"ID":"d07773c9-4cb0-4541-aaca-e1ddb32e915a","Type":"ContainerDied","Data":"0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153"} Sep 15 21:04:32.199042 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.198718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52" event={"ID":"d07773c9-4cb0-4541-aaca-e1ddb32e915a","Type":"ContainerDied","Data":"03d4173c62f8201c38acfa155e91386d84c0563ab1334577a7286ecef01d0c38"} Sep 15 21:04:32.199042 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.198736 2579 scope.go:117] "RemoveContainer" containerID="0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153" Sep 15 21:04:32.242047 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.242020 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52"] Sep 15 21:04:32.247349 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.247326 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-a-466b1a2b-kserve-7677f84cf7-b8p52"] Sep 15 21:04:32.281026 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.280996 2579 scope.go:117] "RemoveContainer" containerID="0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153" Sep 15 21:04:32.281380 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:04:32.281357 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153\": container with ID starting with 0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153 not found: ID does not exist" containerID="0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153" Sep 15 21:04:32.281469 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.281390 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153"} err="failed to get container status \"0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153\": rpc error: code = NotFound desc = could not find container \"0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153\": container with ID starting with 0c2a75eebe20d48599a5e4cf79351f19bc0f1ba0729c423e2d38e49185e2b153 not found: ID does not exist" Sep 15 21:04:32.420426 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.420358 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:04:32.461653 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.461621 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-tmp-dir\") pod \"b73f36f1-fc73-41e3-9143-38d769f632f1\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " Sep 15 21:04:32.461825 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.461700 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-dshm\") pod \"b73f36f1-fc73-41e3-9143-38d769f632f1\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " Sep 15 21:04:32.461825 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.461767 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-model-cache\") pod \"b73f36f1-fc73-41e3-9143-38d769f632f1\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " Sep 15 21:04:32.461957 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.461826 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-home\") pod \"b73f36f1-fc73-41e3-9143-38d769f632f1\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " Sep 15 21:04:32.461957 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.461881 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73f36f1-fc73-41e3-9143-38d769f632f1-tls-certs\") pod \"b73f36f1-fc73-41e3-9143-38d769f632f1\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " Sep 15 21:04:32.461957 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.461899 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b73f36f1-fc73-41e3-9143-38d769f632f1" (UID: "b73f36f1-fc73-41e3-9143-38d769f632f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:04:32.461957 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.461922 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2fqn\" (UniqueName: \"kubernetes.io/projected/b73f36f1-fc73-41e3-9143-38d769f632f1-kube-api-access-t2fqn\") pod \"b73f36f1-fc73-41e3-9143-38d769f632f1\" (UID: \"b73f36f1-fc73-41e3-9143-38d769f632f1\") " Sep 15 21:04:32.462164 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.462027 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b73f36f1-fc73-41e3-9143-38d769f632f1" (UID: "b73f36f1-fc73-41e3-9143-38d769f632f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:04:32.462164 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.462056 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-home" (OuterVolumeSpecName: "home") pod "b73f36f1-fc73-41e3-9143-38d769f632f1" (UID: "b73f36f1-fc73-41e3-9143-38d769f632f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:04:32.462369 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.462347 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:32.462424 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.462378 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:32.462479 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.462395 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:32.464158 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.464136 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-dshm" (OuterVolumeSpecName: "dshm") pod "b73f36f1-fc73-41e3-9143-38d769f632f1" (UID: "b73f36f1-fc73-41e3-9143-38d769f632f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:04:32.464580 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.464556 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73f36f1-fc73-41e3-9143-38d769f632f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b73f36f1-fc73-41e3-9143-38d769f632f1" (UID: "b73f36f1-fc73-41e3-9143-38d769f632f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:04:32.464663 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.464598 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73f36f1-fc73-41e3-9143-38d769f632f1-kube-api-access-t2fqn" (OuterVolumeSpecName: "kube-api-access-t2fqn") pod "b73f36f1-fc73-41e3-9143-38d769f632f1" (UID: "b73f36f1-fc73-41e3-9143-38d769f632f1"). InnerVolumeSpecName "kube-api-access-t2fqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:04:32.563492 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.563441 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73f36f1-fc73-41e3-9143-38d769f632f1-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:32.563492 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.563489 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2fqn\" (UniqueName: \"kubernetes.io/projected/b73f36f1-fc73-41e3-9143-38d769f632f1-kube-api-access-t2fqn\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:32.563718 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:32.563534 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73f36f1-fc73-41e3-9143-38d769f632f1-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:04:33.206065 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.206028 2579 generic.go:356] "Generic (PLEG): container finished" podID="b73f36f1-fc73-41e3-9143-38d769f632f1" containerID="cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6" exitCode=0 Sep 15 21:04:33.206587 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.206075 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" Sep 15 21:04:33.206587 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.206117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" event={"ID":"b73f36f1-fc73-41e3-9143-38d769f632f1","Type":"ContainerDied","Data":"cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6"} Sep 15 21:04:33.206587 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.206167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp" event={"ID":"b73f36f1-fc73-41e3-9143-38d769f632f1","Type":"ContainerDied","Data":"d66db213f7acae8076a3295f4b1100d14cec3dbdb6adbd8c3685d21186e8bbae"} Sep 15 21:04:33.206587 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.206192 2579 scope.go:117] "RemoveContainer" containerID="cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6" Sep 15 21:04:33.216246 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.216228 2579 scope.go:117] "RemoveContainer" containerID="cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6" Sep 15 21:04:33.216588 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:04:33.216567 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6\": container with ID starting with cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6 not found: ID does not exist" containerID="cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6" Sep 15 21:04:33.216652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.216598 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6"} err="failed to get container status \"cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6\": rpc error: code = NotFound desc = could not find container \"cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6\": container with ID starting with cf4dce3b07ec499cb95db3dafc69bced992d49ee816eacc8b91a6bb4c36457c6 not found: ID does not exist" Sep 15 21:04:33.227080 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.227054 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp"] Sep 15 21:04:33.230775 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.230752 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-65001159/e2e-lifecycle-model-65001159-kserve-5f64d97f47-97dzp"] Sep 15 21:04:33.311767 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.311731 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b73f36f1-fc73-41e3-9143-38d769f632f1" path="/var/lib/kubelet/pods/b73f36f1-fc73-41e3-9143-38d769f632f1/volumes" Sep 15 21:04:33.312177 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:33.312158 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d07773c9-4cb0-4541-aaca-e1ddb32e915a" path="/var/lib/kubelet/pods/d07773c9-4cb0-4541-aaca-e1ddb32e915a/volumes" Sep 15 21:04:47.477438 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.477394 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr"] Sep 15 21:04:47.477978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.477840 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e6587a-a3c8-4352-8871-7ae10c1e7e58" containerName="payload-processing" Sep 15 21:04:47.477978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.477853 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e6587a-a3c8-4352-8871-7ae10c1e7e58" containerName="payload-processing" Sep 15 21:04:47.477978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.477865 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b73f36f1-fc73-41e3-9143-38d769f632f1" containerName="main" Sep 15 21:04:47.477978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.477870 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b73f36f1-fc73-41e3-9143-38d769f632f1" containerName="main" Sep 15 21:04:47.477978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.477880 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1f0d88-e092-45e5-ab39-ced258c11b50" containerName="revoke-keys" Sep 15 21:04:47.477978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.477887 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1f0d88-e092-45e5-ab39-ced258c11b50" containerName="revoke-keys" Sep 15 21:04:47.477978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.477914 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d07773c9-4cb0-4541-aaca-e1ddb32e915a" containerName="main" Sep 15 21:04:47.477978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.477923 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d07773c9-4cb0-4541-aaca-e1ddb32e915a" containerName="main" Sep 15 21:04:47.478320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.478000 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b73f36f1-fc73-41e3-9143-38d769f632f1" containerName="main" Sep 15 21:04:47.478320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.478008 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d07773c9-4cb0-4541-aaca-e1ddb32e915a" containerName="main" Sep 15 21:04:47.478320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.478014 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e6587a-a3c8-4352-8871-7ae10c1e7e58" containerName="payload-processing" Sep 15 21:04:47.478320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.478020 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1f0d88-e092-45e5-ab39-ced258c11b50" containerName="revoke-keys" Sep 15 21:04:47.480991 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.480975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.483352 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.483332 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"distinct-w4-16bde4-kserve-self-signed-certs\"" Sep 15 21:04:47.493419 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.493392 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr"] Sep 15 21:04:47.613888 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.613857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-home\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.614069 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.613903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03557738-ca55-4c4d-aa1a-ef659efe0f40-tls-certs\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.614069 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.613966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6mz\" (UniqueName: \"kubernetes.io/projected/03557738-ca55-4c4d-aa1a-ef659efe0f40-kube-api-access-sq6mz\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.614069 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.614019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-dshm\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.614069 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.614052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-model-cache\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.614069 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.614068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-tmp-dir\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.715309 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.715268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6mz\" (UniqueName: \"kubernetes.io/projected/03557738-ca55-4c4d-aa1a-ef659efe0f40-kube-api-access-sq6mz\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.715550 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.715321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-dshm\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.715550 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.715443 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-model-cache\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.715550 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.715474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-tmp-dir\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.715741 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.715596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-home\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.715741 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.715651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03557738-ca55-4c4d-aa1a-ef659efe0f40-tls-certs\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.715859 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.715836 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-model-cache\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.715919 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.715897 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-tmp-dir\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.715975 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.715959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-home\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.717712 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.717681 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-dshm\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.718079 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.718064 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03557738-ca55-4c4d-aa1a-ef659efe0f40-tls-certs\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.722863 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.722839 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6mz\" (UniqueName: \"kubernetes.io/projected/03557738-ca55-4c4d-aa1a-ef659efe0f40-kube-api-access-sq6mz\") pod \"distinct-w4-16bde4-kserve-64c4998bcf-l74dr\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:47.790561 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:47.790531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:48.063803 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:48.063778 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr"] Sep 15 21:04:48.065907 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:04:48.065878 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03557738_ca55_4c4d_aa1a_ef659efe0f40.slice/crio-50213034a27da97392ee2801ca66967134de8f1febccb5bdadc96f810e89a420 WatchSource:0}: Error finding container 50213034a27da97392ee2801ca66967134de8f1febccb5bdadc96f810e89a420: Status 404 returned error can't find the container with id 50213034a27da97392ee2801ca66967134de8f1febccb5bdadc96f810e89a420 Sep 15 21:04:48.291456 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:48.291411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" event={"ID":"03557738-ca55-4c4d-aa1a-ef659efe0f40","Type":"ContainerStarted","Data":"2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9"} Sep 15 21:04:48.291456 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:48.291461 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" event={"ID":"03557738-ca55-4c4d-aa1a-ef659efe0f40","Type":"ContainerStarted","Data":"50213034a27da97392ee2801ca66967134de8f1febccb5bdadc96f810e89a420"} Sep 15 21:04:48.306453 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:48.306399 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" podStartSLOduration=1.3063803489999999 podStartE2EDuration="1.306380349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:04:48.306344923 +0000 UTC m=+1685.500298379" watchObservedRunningTime="2026-09-15 21:04:48.306380349 +0000 UTC m=+1685.500333803" Sep 15 21:04:57.791017 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:57.790979 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:57.791426 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:57.791029 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:57.804102 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:57.804073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:04:58.360313 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:04:58.360287 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:05:32.015749 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:32.015714 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g"] Sep 15 21:05:32.016679 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:32.015926 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" containerID="cri-o://c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63" gracePeriod=30 Sep 15 21:05:37.269650 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.269622 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:05:37.309339 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.309242 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istiod-ca-cert\") pod \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " Sep 15 21:05:37.309339 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.309292 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-certs\") pod \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " Sep 15 21:05:37.309339 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.309323 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-podinfo\") pod \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.309661 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-data\") pod \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.309725 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" (UID: "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.309745 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-credential-socket\") pod \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.309799 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-socket\") pod \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.309833 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8jg8\" (UniqueName: \"kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-kube-api-access-v8jg8\") pod \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.309910 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" (UID: "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.310022 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-envoy\") pod \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.310066 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-token\") pod \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\" (UID: \"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7\") " Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.310086 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" (UID: "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.310333 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" (UID: "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.310546 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.310566 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.310579 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.310600 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:05:37.311129 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.311014 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" (UID: "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:05:37.312711 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.312678 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" (UID: "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:05:37.313333 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.313299 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-kube-api-access-v8jg8" (OuterVolumeSpecName: "kube-api-access-v8jg8") pod "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" (UID: "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7"). InnerVolumeSpecName "kube-api-access-v8jg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:05:37.313620 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.313595 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" (UID: "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:05:37.314717 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.314693 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" (UID: "5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:05:37.412095 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.412062 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:05:37.412095 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.412090 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:05:37.412095 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.412098 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:05:37.412333 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.412109 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:05:37.412333 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.412118 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8jg8\" (UniqueName: \"kubernetes.io/projected/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7-kube-api-access-v8jg8\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:05:37.567298 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.567201 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerID="c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63" exitCode=0 Sep 15 21:05:37.567298 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.567254 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" Sep 15 21:05:37.567298 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.567270 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" event={"ID":"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7","Type":"ContainerDied","Data":"c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63"} Sep 15 21:05:37.567686 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.567325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g" event={"ID":"5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7","Type":"ContainerDied","Data":"39939df3002fc1caa629caf6bada99733df83c2900450d869ee59e8c4e26e49a"} Sep 15 21:05:37.567686 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.567347 2579 scope.go:117] "RemoveContainer" containerID="c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63" Sep 15 21:05:37.577540 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.577517 2579 scope.go:117] "RemoveContainer" containerID="9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263" Sep 15 21:05:37.586299 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.586283 2579 scope.go:117] "RemoveContainer" containerID="c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63" Sep 15 21:05:37.586569 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:05:37.586551 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63\": container with ID starting with c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63 not found: ID does not exist" containerID="c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63" Sep 15 21:05:37.586661 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.586582 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63"} err="failed to get container status \"c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63\": rpc error: code = NotFound desc = could not find container \"c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63\": container with ID starting with c355738e602f3f510225559381623cf4f6f61093bcdc533efd5438867fcf3f63 not found: ID does not exist" Sep 15 21:05:37.586661 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.586603 2579 scope.go:117] "RemoveContainer" containerID="9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263" Sep 15 21:05:37.586849 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:05:37.586832 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263\": container with ID starting with 9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263 not found: ID does not exist" containerID="9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263" Sep 15 21:05:37.586889 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.586855 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263"} err="failed to get container status \"9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263\": rpc error: code = NotFound desc = could not find container \"9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263\": container with ID starting with 9ef1cdeb4c56500b16b702ec39a2525ca5088f39d85080d3374243e8337e9263 not found: ID does not exist" Sep 15 21:05:37.596975 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.596952 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g"] Sep 15 21:05:37.603193 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:37.603172 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-65001159-openshift-default-7c6969ddcb-ftf4g"] Sep 15 21:05:39.312366 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:39.312328 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" path="/var/lib/kubelet/pods/5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7/volumes" Sep 15 21:05:43.592700 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:43.592665 2579 scope.go:117] "RemoveContainer" containerID="3dabdd57b7e4e53bdb348ff4936c9c6ac2c2b82f6fd120231ca94733dd890954" Sep 15 21:05:43.602696 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:05:43.602674 2579 scope.go:117] "RemoveContainer" containerID="5b2c573e316dfbaa7db75b7c1d74112cf3e6a2f823ed1900bdfeb82e3241ca93" Sep 15 21:06:03.186328 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.186290 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz"] Sep 15 21:06:03.186917 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.186897 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" Sep 15 21:06:03.186993 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.186920 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" Sep 15 21:06:03.186993 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.186932 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" Sep 15 21:06:03.186993 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.186942 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" Sep 15 21:06:03.187153 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.187062 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" Sep 15 21:06:03.187153 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.187074 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ab52b61-ddc4-457a-9ffd-7a895e3ab9e7" containerName="istio-proxy" Sep 15 21:06:03.190487 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.190458 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.193231 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.193208 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"distinct-2-w4-16bde4-kserve-self-signed-certs\"" Sep 15 21:06:03.200040 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.200018 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz"] Sep 15 21:06:03.266186 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.266154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tmp-dir\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.266384 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.266210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-dshm\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.266384 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.266250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tls-certs\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.266384 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.266299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xf9j\" (UniqueName: \"kubernetes.io/projected/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-kube-api-access-8xf9j\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.266384 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.266323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-model-cache\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.266384 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.266356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-home\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.367440 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.367404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tmp-dir\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.367440 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.367443 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-dshm\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.367696 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.367461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tls-certs\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.367696 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.367518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xf9j\" (UniqueName: \"kubernetes.io/projected/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-kube-api-access-8xf9j\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.367696 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.367540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-model-cache\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.367696 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.367558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-home\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.367909 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.367854 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tmp-dir\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.367992 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.367971 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-home\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.368051 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.368000 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-model-cache\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.370006 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.369979 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-dshm\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.370250 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.370228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tls-certs\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.376177 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.376153 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xf9j\" (UniqueName: \"kubernetes.io/projected/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-kube-api-access-8xf9j\") pod \"distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.500823 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.500741 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:03.774221 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:03.774187 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz"] Sep 15 21:06:03.777565 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:06:03.777535 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27f9970b_ed43_4a9a_a579_e17ea3b1c5b0.slice/crio-938dbda31e495479bd23cd16da41da5f4a95c0fd2d44beea95a65ed1aa49b661 WatchSource:0}: Error finding container 938dbda31e495479bd23cd16da41da5f4a95c0fd2d44beea95a65ed1aa49b661: Status 404 returned error can't find the container with id 938dbda31e495479bd23cd16da41da5f4a95c0fd2d44beea95a65ed1aa49b661 Sep 15 21:06:04.713334 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:04.713291 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" event={"ID":"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0","Type":"ContainerStarted","Data":"76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f"} Sep 15 21:06:04.713334 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:04.713340 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" event={"ID":"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0","Type":"ContainerStarted","Data":"938dbda31e495479bd23cd16da41da5f4a95c0fd2d44beea95a65ed1aa49b661"} Sep 15 21:06:04.733834 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:04.733785 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" podStartSLOduration=1.73376843 podStartE2EDuration="1.73376843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:06:04.732230049 +0000 UTC m=+1761.926183507" watchObservedRunningTime="2026-09-15 21:06:04.73376843 +0000 UTC m=+1761.927721886" Sep 15 21:06:13.501112 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:13.501073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:13.501112 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:13.501119 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:13.514579 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:13.514550 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:13.771764 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:13.771737 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:06:23.464332 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.464287 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c"] Sep 15 21:06:23.468037 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.468015 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.470380 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.470360 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"embedding-w0-3f1c93-kserve-self-signed-certs\"" Sep 15 21:06:23.476456 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.476432 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c"] Sep 15 21:06:23.569666 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.569631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snrhc\" (UniqueName: \"kubernetes.io/projected/9edcd1ee-fc26-43df-bce7-d7226a763c25-kube-api-access-snrhc\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.569860 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.569680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9edcd1ee-fc26-43df-bce7-d7226a763c25-tls-certs\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.569860 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.569714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-tmp-dir\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.569860 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.569773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-dshm\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.569860 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.569815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-model-cache\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.569860 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.569857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-home\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.671112 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.671068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9edcd1ee-fc26-43df-bce7-d7226a763c25-tls-certs\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.671317 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.671131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-tmp-dir\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.671317 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.671174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-dshm\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.671317 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.671245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-model-cache\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.671317 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.671308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-home\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.671582 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.671395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snrhc\" (UniqueName: \"kubernetes.io/projected/9edcd1ee-fc26-43df-bce7-d7226a763c25-kube-api-access-snrhc\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.671654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.671601 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-tmp-dir\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.671711 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.671685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-model-cache\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.672087 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.672059 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-home\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.673609 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.673590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-dshm\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.673870 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.673852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9edcd1ee-fc26-43df-bce7-d7226a763c25-tls-certs\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.685042 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.685019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snrhc\" (UniqueName: \"kubernetes.io/projected/9edcd1ee-fc26-43df-bce7-d7226a763c25-kube-api-access-snrhc\") pod \"embedding-w0-3f1c93-kserve-55f554dccf-2rc6c\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:23.779071 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:23.779040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:24.055179 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:24.055150 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c"] Sep 15 21:06:24.056664 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:06:24.056636 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9edcd1ee_fc26_43df_bce7_d7226a763c25.slice/crio-4bef728ee3f1ebd74f9fb797c3a27f1eec68bae94972177db36018401687a215 WatchSource:0}: Error finding container 4bef728ee3f1ebd74f9fb797c3a27f1eec68bae94972177db36018401687a215: Status 404 returned error can't find the container with id 4bef728ee3f1ebd74f9fb797c3a27f1eec68bae94972177db36018401687a215 Sep 15 21:06:24.824697 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:24.824660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" event={"ID":"9edcd1ee-fc26-43df-bce7-d7226a763c25","Type":"ContainerStarted","Data":"2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2"} Sep 15 21:06:24.825114 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:24.824707 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" event={"ID":"9edcd1ee-fc26-43df-bce7-d7226a763c25","Type":"ContainerStarted","Data":"4bef728ee3f1ebd74f9fb797c3a27f1eec68bae94972177db36018401687a215"} Sep 15 21:06:24.841063 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:24.841015 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" podStartSLOduration=1.840998895 podStartE2EDuration="1.840998895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:06:24.839789908 +0000 UTC m=+1782.033743362" watchObservedRunningTime="2026-09-15 21:06:24.840998895 +0000 UTC m=+1782.034952349" Sep 15 21:06:27.069074 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.069037 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr"] Sep 15 21:06:27.072877 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.072856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.075069 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.075045 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-178002-openshift-default-dockercfg-4crff\"" Sep 15 21:06:27.084382 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.084357 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr"] Sep 15 21:06:27.205014 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.204974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-envoy\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.205205 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.205041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-credential-socket\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.205205 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.205081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-certs\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.205205 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.205189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w99jl\" (UniqueName: \"kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-kube-api-access-w99jl\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.205444 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.205247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a02c7a5-84eb-4469-b758-1e238310af13-istiod-ca-cert\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.205444 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.205337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a02c7a5-84eb-4469-b758-1e238310af13-istio-podinfo\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.205444 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.205387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-data\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.205444 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.205433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-socket\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.205682 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.205463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-istio-token\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306346 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a02c7a5-84eb-4469-b758-1e238310af13-istiod-ca-cert\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306583 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a02c7a5-84eb-4469-b758-1e238310af13-istio-podinfo\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306583 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-data\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306583 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-socket\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306583 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-istio-token\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306583 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-envoy\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306583 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-credential-socket\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306911 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-certs\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306911 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w99jl\" (UniqueName: \"kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-kube-api-access-w99jl\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306911 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306811 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-data\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.306911 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.306840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-socket\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.307092 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.307080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-credential-socket\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.307130 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.307104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a02c7a5-84eb-4469-b758-1e238310af13-istiod-ca-cert\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.307174 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.307149 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-certs\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.309400 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.309371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-envoy\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.309643 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.309624 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a02c7a5-84eb-4469-b758-1e238310af13-istio-podinfo\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.314535 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.314486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-istio-token\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.314658 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.314606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w99jl\" (UniqueName: \"kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-kube-api-access-w99jl\") pod \"e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.386133 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.386037 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:27.666127 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.666030 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr"] Sep 15 21:06:27.672014 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:06:27.671988 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a02c7a5_84eb_4469_b758_1e238310af13.slice/crio-7aefb1a9ab21c2110d0894a20bf5b103c67671e615a3ef17674981bfd7339899 WatchSource:0}: Error finding container 7aefb1a9ab21c2110d0894a20bf5b103c67671e615a3ef17674981bfd7339899: Status 404 returned error can't find the container with id 7aefb1a9ab21c2110d0894a20bf5b103c67671e615a3ef17674981bfd7339899 Sep 15 21:06:27.674325 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.674291 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:06:27.674426 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.674372 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:06:27.674426 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.674414 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:06:27.843186 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.843149 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" event={"ID":"7a02c7a5-84eb-4469-b758-1e238310af13","Type":"ContainerStarted","Data":"0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d"} Sep 15 21:06:27.843360 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.843215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" event={"ID":"7a02c7a5-84eb-4469-b758-1e238310af13","Type":"ContainerStarted","Data":"7aefb1a9ab21c2110d0894a20bf5b103c67671e615a3ef17674981bfd7339899"} Sep 15 21:06:27.863227 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:27.863170 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" podStartSLOduration=0.863150473 podStartE2EDuration="863.150473ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:06:27.861334919 +0000 UTC m=+1785.055288375" watchObservedRunningTime="2026-09-15 21:06:27.863150473 +0000 UTC m=+1785.057103928" Sep 15 21:06:28.386714 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:28.386675 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:28.392212 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:28.392182 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:28.848517 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:28.848466 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:28.849693 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:28.849674 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:33.779554 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:33.779509 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:33.779554 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:33.779559 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:33.792026 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:33.792001 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:33.887844 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:33.887807 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:06:43.633986 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:43.633951 2579 scope.go:117] "RemoveContainer" containerID="ecb0e09b96da4b5022c07c6d5078d0454d94dd93081ca2802d3c1cb4dda7332c" Sep 15 21:06:56.002794 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:56.002759 2579 generic.go:356] "Generic (PLEG): container finished" podID="7a02c7a5-84eb-4469-b758-1e238310af13" containerID="0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d" exitCode=0 Sep 15 21:06:56.003211 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:56.002833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" event={"ID":"7a02c7a5-84eb-4469-b758-1e238310af13","Type":"ContainerDied","Data":"0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d"} Sep 15 21:06:56.003277 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:56.003261 2579 scope.go:117] "RemoveContainer" containerID="0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d" Sep 15 21:06:56.004484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:56.004453 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:06:56.004598 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:56.004549 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:06:56.004598 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:56.004580 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:06:57.010917 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:57.010880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" event={"ID":"7a02c7a5-84eb-4469-b758-1e238310af13","Type":"ContainerStarted","Data":"55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3"} Sep 15 21:06:57.386555 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:57.386484 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:57.388133 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:57.388100 2579 patch_prober.go:28] interesting pod/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:06:57.388287 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:57.388167 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 15 21:06:58.386924 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:58.386881 2579 patch_prober.go:28] interesting pod/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:06:58.387454 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:58.386956 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 15 21:06:58.848673 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:58.848635 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:06:59.386800 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:59.386757 2579 patch_prober.go:28] interesting pod/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:06:59.387026 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:06:59.386835 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 15 21:07:00.386731 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:00.386692 2579 patch_prober.go:28] interesting pod/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:07:00.387152 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:00.387117 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 15 21:07:00.906524 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:00.896667 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g"] Sep 15 21:07:00.911347 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:00.911318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:00.913598 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:00.913565 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g"] Sep 15 21:07:00.913959 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:00.913939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-14aeb7-openshift-default-dockercfg-7l45m\"" Sep 15 21:07:01.061481 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.061444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65pp5\" (UniqueName: \"kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-kube-api-access-65pp5\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.061681 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.061489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-socket\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.061681 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.061589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-envoy\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.061759 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.061709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-certs\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.061759 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.061740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-credential-socket\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.061834 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.061803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-podinfo\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.061834 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.061824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-token\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.061900 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.061862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-data\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.061900 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.061892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istiod-ca-cert\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.162985 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.162886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-certs\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.162985 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.162932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-credential-socket\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.162985 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.162966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-podinfo\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.162985 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.162982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-token\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.163320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.163003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-data\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.163320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.163021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istiod-ca-cert\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.163320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.163201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65pp5\" (UniqueName: \"kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-kube-api-access-65pp5\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.163320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.163255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-socket\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.163699 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.163335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-envoy\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.163699 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.163371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-certs\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.163699 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.163433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-data\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.163699 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.163617 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-credential-socket\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.163889 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.163759 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-socket\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.163889 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.163787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istiod-ca-cert\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.165751 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.165726 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-podinfo\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.165904 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.165886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-envoy\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.170378 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.170355 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-token\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.170629 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.170607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65pp5\" (UniqueName: \"kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-kube-api-access-65pp5\") pod \"e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.226039 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.226009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:01.387410 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.386993 2579 patch_prober.go:28] interesting pod/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:07:01.387410 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.387055 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 15 21:07:01.529449 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.529417 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g"] Sep 15 21:07:01.531575 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:07:01.531485 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5ae23d_ac4a_4969_8c0d_e8f788e865f2.slice/crio-81204a932ef1565f093a12422f46391c8ccffa690af88442f603da99279e8c2c WatchSource:0}: Error finding container 81204a932ef1565f093a12422f46391c8ccffa690af88442f603da99279e8c2c: Status 404 returned error can't find the container with id 81204a932ef1565f093a12422f46391c8ccffa690af88442f603da99279e8c2c Sep 15 21:07:01.534061 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.534025 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:07:01.534175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.534106 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:07:01.534175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:01.534152 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:07:02.040837 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:02.040803 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" event={"ID":"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2","Type":"ContainerStarted","Data":"9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6"} Sep 15 21:07:02.041025 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:02.040855 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" event={"ID":"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2","Type":"ContainerStarted","Data":"81204a932ef1565f093a12422f46391c8ccffa690af88442f603da99279e8c2c"} Sep 15 21:07:02.068041 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:02.067982 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" podStartSLOduration=2.067962712 podStartE2EDuration="2.067962712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:07:02.065815154 +0000 UTC m=+1819.259768609" watchObservedRunningTime="2026-09-15 21:07:02.067962712 +0000 UTC m=+1819.261916167" Sep 15 21:07:02.227118 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:02.227087 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:02.232487 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:02.232464 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:02.386960 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:02.386854 2579 patch_prober.go:28] interesting pod/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:07:02.386960 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:02.386924 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 15 21:07:03.046731 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:03.046698 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:03.047785 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:03.047764 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:03.391207 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:03.391128 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:07:03.392081 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:03.392058 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:07:10.534865 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.534833 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh"] Sep 15 21:07:10.538874 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.538853 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:07:10.541684 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.541661 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-14aeb7-dockercfg-zflsd\"" Sep 15 21:07:10.542221 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.542200 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-14aeb7\"" Sep 15 21:07:10.553740 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.553720 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh"] Sep 15 21:07:10.669462 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.669417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jpng\" (UniqueName: \"kubernetes.io/projected/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-kube-api-access-5jpng\") pod \"payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh\" (UID: \"d08853ba-5b6c-41bd-8a0f-a519c91cad4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:07:10.669462 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.669465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh\" (UID: \"d08853ba-5b6c-41bd-8a0f-a519c91cad4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:07:10.771195 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.771154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jpng\" (UniqueName: \"kubernetes.io/projected/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-kube-api-access-5jpng\") pod \"payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh\" (UID: \"d08853ba-5b6c-41bd-8a0f-a519c91cad4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:07:10.771195 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.771196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh\" (UID: \"d08853ba-5b6c-41bd-8a0f-a519c91cad4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:07:10.772013 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.771992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh\" (UID: \"d08853ba-5b6c-41bd-8a0f-a519c91cad4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:07:10.780283 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.780258 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jpng\" (UniqueName: \"kubernetes.io/projected/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-kube-api-access-5jpng\") pod \"payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh\" (UID: \"d08853ba-5b6c-41bd-8a0f-a519c91cad4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:07:10.849955 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:10.849924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:07:11.123385 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:11.123357 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh"] Sep 15 21:07:11.125084 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:07:11.125056 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08853ba_5b6c_41bd_8a0f_a519c91cad4a.slice/crio-e841d5dea3f78ded736a203a54fbdf29d2d11ba9a4a9cc4a05787e8770454f3b WatchSource:0}: Error finding container e841d5dea3f78ded736a203a54fbdf29d2d11ba9a4a9cc4a05787e8770454f3b: Status 404 returned error can't find the container with id e841d5dea3f78ded736a203a54fbdf29d2d11ba9a4a9cc4a05787e8770454f3b Sep 15 21:07:12.100738 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:12.100696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" event={"ID":"d08853ba-5b6c-41bd-8a0f-a519c91cad4a","Type":"ContainerStarted","Data":"fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e"} Sep 15 21:07:12.101148 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:12.100783 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" event={"ID":"d08853ba-5b6c-41bd-8a0f-a519c91cad4a","Type":"ContainerStarted","Data":"e841d5dea3f78ded736a203a54fbdf29d2d11ba9a4a9cc4a05787e8770454f3b"} Sep 15 21:07:12.101148 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:12.100805 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:07:12.118746 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:12.118695 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" podStartSLOduration=2.118676988 podStartE2EDuration="2.118676988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:07:12.116258293 +0000 UTC m=+1829.310211743" watchObservedRunningTime="2026-09-15 21:07:12.118676988 +0000 UTC m=+1829.312630443" Sep 15 21:07:23.108446 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:23.108405 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:07:37.254668 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:37.254629 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq"] Sep 15 21:07:37.258796 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:37.258777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" Sep 15 21:07:37.261203 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:37.261181 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:07:37.266420 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:37.266395 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq"] Sep 15 21:07:37.360985 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:37.360929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eefc537-8d46-4b0c-8e93-06ed158b6e11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-178002-tkkqq\" (UID: \"5eefc537-8d46-4b0c-8e93-06ed158b6e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" Sep 15 21:07:37.461590 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:37.461553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eefc537-8d46-4b0c-8e93-06ed158b6e11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-178002-tkkqq\" (UID: \"5eefc537-8d46-4b0c-8e93-06ed158b6e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" Sep 15 21:07:37.462175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:37.462155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eefc537-8d46-4b0c-8e93-06ed158b6e11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-178002-tkkqq\" (UID: \"5eefc537-8d46-4b0c-8e93-06ed158b6e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" Sep 15 21:07:37.572115 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:37.572082 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" Sep 15 21:07:37.856154 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:37.856123 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq"] Sep 15 21:07:37.857091 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:07:37.857072 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eefc537_8d46_4b0c_8e93_06ed158b6e11.slice/crio-0463b090701b5672b5d73c22622b3d9b7a4ad074d9e98f9f2970fb07ef969d54 WatchSource:0}: Error finding container 0463b090701b5672b5d73c22622b3d9b7a4ad074d9e98f9f2970fb07ef969d54: Status 404 returned error can't find the container with id 0463b090701b5672b5d73c22622b3d9b7a4ad074d9e98f9f2970fb07ef969d54 Sep 15 21:07:38.249798 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:38.249755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" event={"ID":"5eefc537-8d46-4b0c-8e93-06ed158b6e11","Type":"ContainerStarted","Data":"6ea00636c0df9cc93f35a9b2d9ca025feb0cd738842f119b874ad841abcc73d5"} Sep 15 21:07:38.250002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:38.249824 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" event={"ID":"5eefc537-8d46-4b0c-8e93-06ed158b6e11","Type":"ContainerStarted","Data":"0463b090701b5672b5d73c22622b3d9b7a4ad074d9e98f9f2970fb07ef969d54"} Sep 15 21:07:38.267943 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:38.267879 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" podStartSLOduration=1.267859731 podStartE2EDuration="1.267859731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:07:38.267346196 +0000 UTC m=+1855.461299652" watchObservedRunningTime="2026-09-15 21:07:38.267859731 +0000 UTC m=+1855.461813227" Sep 15 21:07:39.256806 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:39.256771 2579 generic.go:356] "Generic (PLEG): container finished" podID="5eefc537-8d46-4b0c-8e93-06ed158b6e11" containerID="6ea00636c0df9cc93f35a9b2d9ca025feb0cd738842f119b874ad841abcc73d5" exitCode=0 Sep 15 21:07:39.256975 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:39.256856 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" event={"ID":"5eefc537-8d46-4b0c-8e93-06ed158b6e11","Type":"ContainerDied","Data":"6ea00636c0df9cc93f35a9b2d9ca025feb0cd738842f119b874ad841abcc73d5"} Sep 15 21:07:40.460870 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:40.460840 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" Sep 15 21:07:40.613540 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:40.613407 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eefc537-8d46-4b0c-8e93-06ed158b6e11-maas-api-service-ca\") pod \"5eefc537-8d46-4b0c-8e93-06ed158b6e11\" (UID: \"5eefc537-8d46-4b0c-8e93-06ed158b6e11\") " Sep 15 21:07:40.613777 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:40.613750 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eefc537-8d46-4b0c-8e93-06ed158b6e11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5eefc537-8d46-4b0c-8e93-06ed158b6e11" (UID: "5eefc537-8d46-4b0c-8e93-06ed158b6e11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:07:40.613852 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:40.613840 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eefc537-8d46-4b0c-8e93-06ed158b6e11-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:07:41.276553 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:41.276484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" event={"ID":"5eefc537-8d46-4b0c-8e93-06ed158b6e11","Type":"ContainerDied","Data":"0463b090701b5672b5d73c22622b3d9b7a4ad074d9e98f9f2970fb07ef969d54"} Sep 15 21:07:41.276770 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:41.276616 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0463b090701b5672b5d73c22622b3d9b7a4ad074d9e98f9f2970fb07ef969d54" Sep 15 21:07:41.276770 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:41.276693 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq" Sep 15 21:07:43.655175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:43.655143 2579 scope.go:117] "RemoveContainer" containerID="4436f955620e831db2ae525d46c818d74a18025f231507054cdd9349b6989340" Sep 15 21:07:43.664432 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:43.664412 2579 scope.go:117] "RemoveContainer" containerID="e5e5370eca8be8542fa570e70e880f6236df4112c52c849815e18d01fa7ce26f" Sep 15 21:07:47.320010 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:47.319974 2579 generic.go:356] "Generic (PLEG): container finished" podID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerID="9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6" exitCode=0 Sep 15 21:07:47.320532 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:47.320056 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" event={"ID":"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2","Type":"ContainerDied","Data":"9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6"} Sep 15 21:07:47.320632 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:47.320620 2579 scope.go:117] "RemoveContainer" containerID="9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6" Sep 15 21:07:47.321990 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:47.321963 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:07:47.322066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:47.322052 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:07:47.322105 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:47.322086 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:07:48.047085 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:48.047044 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:48.327423 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:48.327318 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" event={"ID":"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2","Type":"ContainerStarted","Data":"3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2"} Sep 15 21:07:49.226986 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:49.226943 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:49.228463 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:49.228438 2579 patch_prober.go:28] interesting pod/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 21:07:49.228603 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:49.228536 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 21:07:50.226484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:50.226444 2579 patch_prober.go:28] interesting pod/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 21:07:50.227032 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:50.226540 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 21:07:51.226875 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:51.226830 2579 patch_prober.go:28] interesting pod/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 21:07:51.227305 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:51.226900 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 21:07:52.227005 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:52.226962 2579 patch_prober.go:28] interesting pod/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 21:07:52.227459 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:52.227027 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 21:07:52.972344 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:52.972303 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq"] Sep 15 21:07:52.976069 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:52.976036 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-178002-tkkqq"] Sep 15 21:07:53.227328 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:53.227227 2579 patch_prober.go:28] interesting pod/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 21:07:53.227328 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:53.227295 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 21:07:53.312567 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:53.312530 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eefc537-8d46-4b0c-8e93-06ed158b6e11" path="/var/lib/kubelet/pods/5eefc537-8d46-4b0c-8e93-06ed158b6e11/volumes" Sep 15 21:07:54.227425 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:54.227379 2579 patch_prober.go:28] interesting pod/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 21:07:54.228081 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:54.227453 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 21:07:55.230796 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:55.230765 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:55.231209 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:55.231168 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:55.231923 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:55.231902 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:07:59.875094 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:59.875054 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr"] Sep 15 21:07:59.875599 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:07:59.875317 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" containerID="cri-o://55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3" gracePeriod=30 Sep 15 21:08:00.225134 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.225051 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds"] Sep 15 21:08:00.225558 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.225544 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eefc537-8d46-4b0c-8e93-06ed158b6e11" containerName="revoke-keys" Sep 15 21:08:00.225613 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.225560 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eefc537-8d46-4b0c-8e93-06ed158b6e11" containerName="revoke-keys" Sep 15 21:08:00.225653 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.225642 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eefc537-8d46-4b0c-8e93-06ed158b6e11" containerName="revoke-keys" Sep 15 21:08:00.228949 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.228932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" Sep 15 21:08:00.231115 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.231097 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:08:00.238694 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.238666 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds"] Sep 15 21:08:00.324521 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.324460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01302a9f-6c4a-42d6-812e-adc3897ba19e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14aeb7-7njds\" (UID: \"01302a9f-6c4a-42d6-812e-adc3897ba19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" Sep 15 21:08:00.426279 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.426177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01302a9f-6c4a-42d6-812e-adc3897ba19e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14aeb7-7njds\" (UID: \"01302a9f-6c4a-42d6-812e-adc3897ba19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" Sep 15 21:08:00.426901 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.426877 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01302a9f-6c4a-42d6-812e-adc3897ba19e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14aeb7-7njds\" (UID: \"01302a9f-6c4a-42d6-812e-adc3897ba19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" Sep 15 21:08:00.540860 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.540829 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" Sep 15 21:08:00.645583 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.645547 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b"] Sep 15 21:08:00.649255 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.649236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b" Sep 15 21:08:00.659467 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.659440 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b"] Sep 15 21:08:00.730764 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.730720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0327ad-a022-4296-86f9-efe56534d05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b\" (UID: \"7f0327ad-a022-4296-86f9-efe56534d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b" Sep 15 21:08:00.814252 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.814221 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds"] Sep 15 21:08:00.815268 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:08:00.815239 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01302a9f_6c4a_42d6_812e_adc3897ba19e.slice/crio-9c3f33e24ed203253004db3448ddadc7cb903b0fcee57c48539ec07bf88100aa WatchSource:0}: Error finding container 9c3f33e24ed203253004db3448ddadc7cb903b0fcee57c48539ec07bf88100aa: Status 404 returned error can't find the container with id 9c3f33e24ed203253004db3448ddadc7cb903b0fcee57c48539ec07bf88100aa Sep 15 21:08:00.832522 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.832474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0327ad-a022-4296-86f9-efe56534d05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b\" (UID: \"7f0327ad-a022-4296-86f9-efe56534d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b" Sep 15 21:08:00.833057 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.833032 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0327ad-a022-4296-86f9-efe56534d05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b\" (UID: \"7f0327ad-a022-4296-86f9-efe56534d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b" Sep 15 21:08:00.967220 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:00.967176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b" Sep 15 21:08:01.264897 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:01.264869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b"] Sep 15 21:08:01.265306 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:08:01.265283 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0327ad_a022_4296_86f9_efe56534d05e.slice/crio-ebb79ff5c0951f477d7f8635ff2db9c06bf930419b549d10a62126861ef78cb3 WatchSource:0}: Error finding container ebb79ff5c0951f477d7f8635ff2db9c06bf930419b549d10a62126861ef78cb3: Status 404 returned error can't find the container with id ebb79ff5c0951f477d7f8635ff2db9c06bf930419b549d10a62126861ef78cb3 Sep 15 21:08:01.408907 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:01.408817 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b" event={"ID":"7f0327ad-a022-4296-86f9-efe56534d05e","Type":"ContainerStarted","Data":"ebb79ff5c0951f477d7f8635ff2db9c06bf930419b549d10a62126861ef78cb3"} Sep 15 21:08:01.410553 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:01.410520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" event={"ID":"01302a9f-6c4a-42d6-812e-adc3897ba19e","Type":"ContainerStarted","Data":"d955ec0d7c4fdf5f35115212dfdb22fb934e02eb6dd27196a696113f484e3c5c"} Sep 15 21:08:01.410686 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:01.410579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" event={"ID":"01302a9f-6c4a-42d6-812e-adc3897ba19e","Type":"ContainerStarted","Data":"9c3f33e24ed203253004db3448ddadc7cb903b0fcee57c48539ec07bf88100aa"} Sep 15 21:08:01.455487 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:01.455432 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" podStartSLOduration=1.4554139560000001 podStartE2EDuration="1.455413956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:01.451212443 +0000 UTC m=+1878.645165898" watchObservedRunningTime="2026-09-15 21:08:01.455413956 +0000 UTC m=+1878.649367411" Sep 15 21:08:02.418730 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:02.418693 2579 generic.go:356] "Generic (PLEG): container finished" podID="01302a9f-6c4a-42d6-812e-adc3897ba19e" containerID="d955ec0d7c4fdf5f35115212dfdb22fb934e02eb6dd27196a696113f484e3c5c" exitCode=0 Sep 15 21:08:02.419217 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:02.418737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" event={"ID":"01302a9f-6c4a-42d6-812e-adc3897ba19e","Type":"ContainerDied","Data":"d955ec0d7c4fdf5f35115212dfdb22fb934e02eb6dd27196a696113f484e3c5c"} Sep 15 21:08:02.420716 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:02.420688 2579 generic.go:356] "Generic (PLEG): container finished" podID="7f0327ad-a022-4296-86f9-efe56534d05e" containerID="9c6be14c070de37cdf8398b73f6f391872199afdc434b5decc97e0d09fc69dee" exitCode=0 Sep 15 21:08:02.420819 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:02.420735 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b" event={"ID":"7f0327ad-a022-4296-86f9-efe56534d05e","Type":"ContainerDied","Data":"9c6be14c070de37cdf8398b73f6f391872199afdc434b5decc97e0d09fc69dee"} Sep 15 21:08:03.393960 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:03.393908 2579 patch_prober.go:28] interesting pod/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:08:03.394136 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:03.393985 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:08:03.579736 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:03.579711 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b" Sep 15 21:08:03.582731 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:03.582711 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" Sep 15 21:08:03.663912 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:03.663817 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01302a9f-6c4a-42d6-812e-adc3897ba19e-maas-api-service-ca\") pod \"01302a9f-6c4a-42d6-812e-adc3897ba19e\" (UID: \"01302a9f-6c4a-42d6-812e-adc3897ba19e\") " Sep 15 21:08:03.663912 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:03.663895 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0327ad-a022-4296-86f9-efe56534d05e-maas-api-service-ca\") pod \"7f0327ad-a022-4296-86f9-efe56534d05e\" (UID: \"7f0327ad-a022-4296-86f9-efe56534d05e\") " Sep 15 21:08:03.664215 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:03.664191 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01302a9f-6c4a-42d6-812e-adc3897ba19e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01302a9f-6c4a-42d6-812e-adc3897ba19e" (UID: "01302a9f-6c4a-42d6-812e-adc3897ba19e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:03.664257 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:03.664217 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0327ad-a022-4296-86f9-efe56534d05e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f0327ad-a022-4296-86f9-efe56534d05e" (UID: "7f0327ad-a022-4296-86f9-efe56534d05e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:03.765878 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:03.765840 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0327ad-a022-4296-86f9-efe56534d05e-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:03.765878 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:03.765875 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01302a9f-6c4a-42d6-812e-adc3897ba19e-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:04.434857 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:04.434779 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b" Sep 15 21:08:04.434857 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:04.434811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b" event={"ID":"7f0327ad-a022-4296-86f9-efe56534d05e","Type":"ContainerDied","Data":"ebb79ff5c0951f477d7f8635ff2db9c06bf930419b549d10a62126861ef78cb3"} Sep 15 21:08:04.435064 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:04.434863 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebb79ff5c0951f477d7f8635ff2db9c06bf930419b549d10a62126861ef78cb3" Sep 15 21:08:04.436142 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:04.436121 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" event={"ID":"01302a9f-6c4a-42d6-812e-adc3897ba19e","Type":"ContainerDied","Data":"9c3f33e24ed203253004db3448ddadc7cb903b0fcee57c48539ec07bf88100aa"} Sep 15 21:08:04.436271 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:04.436154 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c3f33e24ed203253004db3448ddadc7cb903b0fcee57c48539ec07bf88100aa" Sep 15 21:08:04.436271 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:04.436169 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds" Sep 15 21:08:05.134802 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.134777 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:08:05.278981 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.278950 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a02c7a5-84eb-4469-b758-1e238310af13-istiod-ca-cert\") pod \"7a02c7a5-84eb-4469-b758-1e238310af13\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " Sep 15 21:08:05.279154 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.278993 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-envoy\") pod \"7a02c7a5-84eb-4469-b758-1e238310af13\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " Sep 15 21:08:05.279154 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279029 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-istio-token\") pod \"7a02c7a5-84eb-4469-b758-1e238310af13\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " Sep 15 21:08:05.279154 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279071 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a02c7a5-84eb-4469-b758-1e238310af13-istio-podinfo\") pod \"7a02c7a5-84eb-4469-b758-1e238310af13\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " Sep 15 21:08:05.279154 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279124 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-socket\") pod \"7a02c7a5-84eb-4469-b758-1e238310af13\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " Sep 15 21:08:05.279358 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279155 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w99jl\" (UniqueName: \"kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-kube-api-access-w99jl\") pod \"7a02c7a5-84eb-4469-b758-1e238310af13\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " Sep 15 21:08:05.279358 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279209 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-credential-socket\") pod \"7a02c7a5-84eb-4469-b758-1e238310af13\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " Sep 15 21:08:05.279358 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279262 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-certs\") pod \"7a02c7a5-84eb-4469-b758-1e238310af13\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " Sep 15 21:08:05.279358 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279350 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-data\") pod \"7a02c7a5-84eb-4469-b758-1e238310af13\" (UID: \"7a02c7a5-84eb-4469-b758-1e238310af13\") " Sep 15 21:08:05.279605 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279384 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a02c7a5-84eb-4469-b758-1e238310af13-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a02c7a5-84eb-4469-b758-1e238310af13" (UID: "7a02c7a5-84eb-4469-b758-1e238310af13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:05.279605 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279451 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a02c7a5-84eb-4469-b758-1e238310af13" (UID: "7a02c7a5-84eb-4469-b758-1e238310af13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:05.279711 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279658 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a02c7a5-84eb-4469-b758-1e238310af13" (UID: "7a02c7a5-84eb-4469-b758-1e238310af13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:05.279711 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279669 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a02c7a5-84eb-4469-b758-1e238310af13" (UID: "7a02c7a5-84eb-4469-b758-1e238310af13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:05.279711 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.279685 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a02c7a5-84eb-4469-b758-1e238310af13" (UID: "7a02c7a5-84eb-4469-b758-1e238310af13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:05.280044 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.280024 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:05.280096 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.280049 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a02c7a5-84eb-4469-b758-1e238310af13-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:05.280096 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.280062 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:05.280096 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.280077 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:05.280096 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.280091 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:05.281513 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.281475 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a02c7a5-84eb-4469-b758-1e238310af13-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a02c7a5-84eb-4469-b758-1e238310af13" (UID: "7a02c7a5-84eb-4469-b758-1e238310af13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:08:05.281883 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.281864 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a02c7a5-84eb-4469-b758-1e238310af13" (UID: "7a02c7a5-84eb-4469-b758-1e238310af13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:05.282111 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.282094 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-kube-api-access-w99jl" (OuterVolumeSpecName: "kube-api-access-w99jl") pod "7a02c7a5-84eb-4469-b758-1e238310af13" (UID: "7a02c7a5-84eb-4469-b758-1e238310af13"). InnerVolumeSpecName "kube-api-access-w99jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:05.282111 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.282098 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a02c7a5-84eb-4469-b758-1e238310af13" (UID: "7a02c7a5-84eb-4469-b758-1e238310af13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:05.381652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.381618 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a02c7a5-84eb-4469-b758-1e238310af13-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:05.381652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.381644 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:05.381652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.381653 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a02c7a5-84eb-4469-b758-1e238310af13-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:05.381886 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.381661 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w99jl\" (UniqueName: \"kubernetes.io/projected/7a02c7a5-84eb-4469-b758-1e238310af13-kube-api-access-w99jl\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:05.443517 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.443465 2579 generic.go:356] "Generic (PLEG): container finished" podID="7a02c7a5-84eb-4469-b758-1e238310af13" containerID="55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3" exitCode=0 Sep 15 21:08:05.443686 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.443523 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" Sep 15 21:08:05.443686 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.443549 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" event={"ID":"7a02c7a5-84eb-4469-b758-1e238310af13","Type":"ContainerDied","Data":"55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3"} Sep 15 21:08:05.443686 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.443604 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr" event={"ID":"7a02c7a5-84eb-4469-b758-1e238310af13","Type":"ContainerDied","Data":"7aefb1a9ab21c2110d0894a20bf5b103c67671e615a3ef17674981bfd7339899"} Sep 15 21:08:05.443686 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.443626 2579 scope.go:117] "RemoveContainer" containerID="55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3" Sep 15 21:08:05.453087 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.453065 2579 scope.go:117] "RemoveContainer" containerID="0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d" Sep 15 21:08:05.461860 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.461840 2579 scope.go:117] "RemoveContainer" containerID="55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3" Sep 15 21:08:05.462145 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:08:05.462123 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3\": container with ID starting with 55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3 not found: ID does not exist" containerID="55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3" Sep 15 21:08:05.462238 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.462153 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3"} err="failed to get container status \"55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3\": rpc error: code = NotFound desc = could not find container \"55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3\": container with ID starting with 55a226e972495958b7d759ed6bc40be0294c6b8c51c785e6f3c9ce9c802b26c3 not found: ID does not exist" Sep 15 21:08:05.462238 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.462175 2579 scope.go:117] "RemoveContainer" containerID="0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d" Sep 15 21:08:05.462542 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:08:05.462515 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d\": container with ID starting with 0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d not found: ID does not exist" containerID="0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d" Sep 15 21:08:05.462643 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.462546 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d"} err="failed to get container status \"0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d\": rpc error: code = NotFound desc = could not find container \"0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d\": container with ID starting with 0a4a482e3cc8c0b2b6af3a6c07132579435cea55cda46bd0ce2407d456fd5f6d not found: ID does not exist" Sep 15 21:08:05.466616 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.466593 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr"] Sep 15 21:08:05.472776 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.472755 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-178002-openshift-default-958bdf7c6-8sjvr"] Sep 15 21:08:05.528872 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.528838 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw"] Sep 15 21:08:05.539090 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:05.539010 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c"] Sep 15 21:08:07.312604 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:07.312570 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" path="/var/lib/kubelet/pods/7a02c7a5-84eb-4469-b758-1e238310af13/volumes" Sep 15 21:08:15.901660 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:15.901624 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl"] Sep 15 21:08:15.907105 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:15.907076 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3f1c93-29825100-q4hkl"] Sep 15 21:08:16.121859 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.121817 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh"] Sep 15 21:08:16.122098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.122057 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" podUID="d08853ba-5b6c-41bd-8a0f-a519c91cad4a" containerName="payload-processing" containerID="cri-o://fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e" gracePeriod=30 Sep 15 21:08:16.166228 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.166143 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b"] Sep 15 21:08:16.176686 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.176655 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3f1c93-cq29b"] Sep 15 21:08:16.391940 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.391913 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:08:16.503002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.502517 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-plugins-config-volume\") pod \"d08853ba-5b6c-41bd-8a0f-a519c91cad4a\" (UID: \"d08853ba-5b6c-41bd-8a0f-a519c91cad4a\") " Sep 15 21:08:16.503002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.502568 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jpng\" (UniqueName: \"kubernetes.io/projected/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-kube-api-access-5jpng\") pod \"d08853ba-5b6c-41bd-8a0f-a519c91cad4a\" (UID: \"d08853ba-5b6c-41bd-8a0f-a519c91cad4a\") " Sep 15 21:08:16.503002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.502899 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d08853ba-5b6c-41bd-8a0f-a519c91cad4a" (UID: "d08853ba-5b6c-41bd-8a0f-a519c91cad4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:16.503002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.502991 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.505550 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.505520 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-kube-api-access-5jpng" (OuterVolumeSpecName: "kube-api-access-5jpng") pod "d08853ba-5b6c-41bd-8a0f-a519c91cad4a" (UID: "d08853ba-5b6c-41bd-8a0f-a519c91cad4a"). InnerVolumeSpecName "kube-api-access-5jpng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:16.509717 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.509692 2579 generic.go:356] "Generic (PLEG): container finished" podID="d08853ba-5b6c-41bd-8a0f-a519c91cad4a" containerID="fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e" exitCode=0 Sep 15 21:08:16.509827 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.509740 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" Sep 15 21:08:16.509827 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.509773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" event={"ID":"d08853ba-5b6c-41bd-8a0f-a519c91cad4a","Type":"ContainerDied","Data":"fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e"} Sep 15 21:08:16.509904 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.509833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh" event={"ID":"d08853ba-5b6c-41bd-8a0f-a519c91cad4a","Type":"ContainerDied","Data":"e841d5dea3f78ded736a203a54fbdf29d2d11ba9a4a9cc4a05787e8770454f3b"} Sep 15 21:08:16.509904 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.509861 2579 scope.go:117] "RemoveContainer" containerID="fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e" Sep 15 21:08:16.522949 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.522925 2579 scope.go:117] "RemoveContainer" containerID="fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e" Sep 15 21:08:16.523251 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:08:16.523232 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e\": container with ID starting with fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e not found: ID does not exist" containerID="fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e" Sep 15 21:08:16.523313 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.523259 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e"} err="failed to get container status \"fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e\": rpc error: code = NotFound desc = could not find container \"fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e\": container with ID starting with fdd48da0fb21d4c6f6940e356dfffee77f5b501359613cdb34843b1421b8457e not found: ID does not exist" Sep 15 21:08:16.537515 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.537463 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh"] Sep 15 21:08:16.543220 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.543186 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14aeb7-7b87b7b4fc-lfzkh"] Sep 15 21:08:16.580284 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.580247 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds"] Sep 15 21:08:16.584921 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.584892 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14aeb7-7njds"] Sep 15 21:08:16.604234 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:16.604188 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jpng\" (UniqueName: \"kubernetes.io/projected/d08853ba-5b6c-41bd-8a0f-a519c91cad4a-kube-api-access-5jpng\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:17.313771 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:17.313738 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01302a9f-6c4a-42d6-812e-adc3897ba19e" path="/var/lib/kubelet/pods/01302a9f-6c4a-42d6-812e-adc3897ba19e/volumes" Sep 15 21:08:17.314152 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:17.314002 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e56f904-6b9a-40bc-8be8-18b61977ba94" path="/var/lib/kubelet/pods/5e56f904-6b9a-40bc-8be8-18b61977ba94/volumes" Sep 15 21:08:17.314253 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:17.314240 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f0327ad-a022-4296-86f9-efe56534d05e" path="/var/lib/kubelet/pods/7f0327ad-a022-4296-86f9-efe56534d05e/volumes" Sep 15 21:08:17.314575 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:17.314563 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08853ba-5b6c-41bd-8a0f-a519c91cad4a" path="/var/lib/kubelet/pods/d08853ba-5b6c-41bd-8a0f-a519c91cad4a/volumes" Sep 15 21:08:20.549424 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.549359 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" podUID="199b0526-668c-4746-8455-dcb606fd015b" containerName="main" containerID="cri-o://e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863" gracePeriod=45 Sep 15 21:08:20.564897 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.564865 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" podUID="9edcd1ee-fc26-43df-bce7-d7226a763c25" containerName="main" containerID="cri-o://2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2" gracePeriod=45 Sep 15 21:08:20.829486 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.829460 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:08:20.853322 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.853294 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:08:20.950873 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.950845 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b0526-668c-4746-8455-dcb606fd015b-tls-certs\") pod \"199b0526-668c-4746-8455-dcb606fd015b\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " Sep 15 21:08:20.951053 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.950894 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-home\") pod \"9edcd1ee-fc26-43df-bce7-d7226a763c25\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " Sep 15 21:08:20.951053 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.950927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-model-cache\") pod \"199b0526-668c-4746-8455-dcb606fd015b\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " Sep 15 21:08:20.951053 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951006 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-model-cache\") pod \"9edcd1ee-fc26-43df-bce7-d7226a763c25\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " Sep 15 21:08:20.951053 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951038 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-dshm\") pod \"9edcd1ee-fc26-43df-bce7-d7226a763c25\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " Sep 15 21:08:20.951264 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951069 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fk2r\" (UniqueName: \"kubernetes.io/projected/199b0526-668c-4746-8455-dcb606fd015b-kube-api-access-4fk2r\") pod \"199b0526-668c-4746-8455-dcb606fd015b\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " Sep 15 21:08:20.951264 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951114 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-tmp-dir\") pod \"199b0526-668c-4746-8455-dcb606fd015b\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " Sep 15 21:08:20.951264 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951144 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-dshm\") pod \"199b0526-668c-4746-8455-dcb606fd015b\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " Sep 15 21:08:20.951264 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951171 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snrhc\" (UniqueName: \"kubernetes.io/projected/9edcd1ee-fc26-43df-bce7-d7226a763c25-kube-api-access-snrhc\") pod \"9edcd1ee-fc26-43df-bce7-d7226a763c25\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " Sep 15 21:08:20.951264 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951216 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-home" (OuterVolumeSpecName: "home") pod "9edcd1ee-fc26-43df-bce7-d7226a763c25" (UID: "9edcd1ee-fc26-43df-bce7-d7226a763c25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.951590 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951223 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-model-cache" (OuterVolumeSpecName: "model-cache") pod "199b0526-668c-4746-8455-dcb606fd015b" (UID: "199b0526-668c-4746-8455-dcb606fd015b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.951590 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951229 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-tmp-dir\") pod \"9edcd1ee-fc26-43df-bce7-d7226a763c25\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " Sep 15 21:08:20.951590 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951293 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-model-cache" (OuterVolumeSpecName: "model-cache") pod "9edcd1ee-fc26-43df-bce7-d7226a763c25" (UID: "9edcd1ee-fc26-43df-bce7-d7226a763c25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.951590 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951343 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-home\") pod \"199b0526-668c-4746-8455-dcb606fd015b\" (UID: \"199b0526-668c-4746-8455-dcb606fd015b\") " Sep 15 21:08:20.951590 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951437 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9edcd1ee-fc26-43df-bce7-d7226a763c25" (UID: "9edcd1ee-fc26-43df-bce7-d7226a763c25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.951590 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951441 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9edcd1ee-fc26-43df-bce7-d7226a763c25-tls-certs\") pod \"9edcd1ee-fc26-43df-bce7-d7226a763c25\" (UID: \"9edcd1ee-fc26-43df-bce7-d7226a763c25\") " Sep 15 21:08:20.952072 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.951602 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "199b0526-668c-4746-8455-dcb606fd015b" (UID: "199b0526-668c-4746-8455-dcb606fd015b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.952072 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.952037 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-home" (OuterVolumeSpecName: "home") pod "199b0526-668c-4746-8455-dcb606fd015b" (UID: "199b0526-668c-4746-8455-dcb606fd015b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.952190 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.952082 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:20.952190 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.952102 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:20.952190 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.952117 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:20.952190 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.952131 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:20.952190 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.952143 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:20.953467 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.953437 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-dshm" (OuterVolumeSpecName: "dshm") pod "9edcd1ee-fc26-43df-bce7-d7226a763c25" (UID: "9edcd1ee-fc26-43df-bce7-d7226a763c25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.953647 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.953490 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199b0526-668c-4746-8455-dcb606fd015b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "199b0526-668c-4746-8455-dcb606fd015b" (UID: "199b0526-668c-4746-8455-dcb606fd015b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:20.953866 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.953851 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-dshm" (OuterVolumeSpecName: "dshm") pod "199b0526-668c-4746-8455-dcb606fd015b" (UID: "199b0526-668c-4746-8455-dcb606fd015b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.954009 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.953991 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edcd1ee-fc26-43df-bce7-d7226a763c25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9edcd1ee-fc26-43df-bce7-d7226a763c25" (UID: "9edcd1ee-fc26-43df-bce7-d7226a763c25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:20.954050 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.954030 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edcd1ee-fc26-43df-bce7-d7226a763c25-kube-api-access-snrhc" (OuterVolumeSpecName: "kube-api-access-snrhc") pod "9edcd1ee-fc26-43df-bce7-d7226a763c25" (UID: "9edcd1ee-fc26-43df-bce7-d7226a763c25"). InnerVolumeSpecName "kube-api-access-snrhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:20.954207 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:20.954190 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199b0526-668c-4746-8455-dcb606fd015b-kube-api-access-4fk2r" (OuterVolumeSpecName: "kube-api-access-4fk2r") pod "199b0526-668c-4746-8455-dcb606fd015b" (UID: "199b0526-668c-4746-8455-dcb606fd015b"). InnerVolumeSpecName "kube-api-access-4fk2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:21.053730 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.053676 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9edcd1ee-fc26-43df-bce7-d7226a763c25-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:21.053730 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.053723 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fk2r\" (UniqueName: \"kubernetes.io/projected/199b0526-668c-4746-8455-dcb606fd015b-kube-api-access-4fk2r\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:21.053730 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.053735 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:21.053730 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.053746 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snrhc\" (UniqueName: \"kubernetes.io/projected/9edcd1ee-fc26-43df-bce7-d7226a763c25-kube-api-access-snrhc\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:21.054019 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.053754 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199b0526-668c-4746-8455-dcb606fd015b-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:21.054019 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.053762 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9edcd1ee-fc26-43df-bce7-d7226a763c25-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:21.054019 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.053771 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b0526-668c-4746-8455-dcb606fd015b-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:21.539820 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.539783 2579 generic.go:356] "Generic (PLEG): container finished" podID="9edcd1ee-fc26-43df-bce7-d7226a763c25" containerID="2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2" exitCode=0 Sep 15 21:08:21.539820 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.539823 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" Sep 15 21:08:21.540072 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.539871 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" event={"ID":"9edcd1ee-fc26-43df-bce7-d7226a763c25","Type":"ContainerDied","Data":"2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2"} Sep 15 21:08:21.540072 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.539917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c" event={"ID":"9edcd1ee-fc26-43df-bce7-d7226a763c25","Type":"ContainerDied","Data":"4bef728ee3f1ebd74f9fb797c3a27f1eec68bae94972177db36018401687a215"} Sep 15 21:08:21.540072 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.539935 2579 scope.go:117] "RemoveContainer" containerID="2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2" Sep 15 21:08:21.541487 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.541462 2579 generic.go:356] "Generic (PLEG): container finished" podID="199b0526-668c-4746-8455-dcb606fd015b" containerID="e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863" exitCode=0 Sep 15 21:08:21.541487 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.541519 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" Sep 15 21:08:21.541487 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.541526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" event={"ID":"199b0526-668c-4746-8455-dcb606fd015b","Type":"ContainerDied","Data":"e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863"} Sep 15 21:08:21.541487 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.541572 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw" event={"ID":"199b0526-668c-4746-8455-dcb606fd015b","Type":"ContainerDied","Data":"82142bba388648c8a31d69b703742654200926765b0f9efa8250c544beae9a10"} Sep 15 21:08:21.549237 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.549222 2579 scope.go:117] "RemoveContainer" containerID="2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2" Sep 15 21:08:21.549554 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:08:21.549537 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2\": container with ID starting with 2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2 not found: ID does not exist" containerID="2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2" Sep 15 21:08:21.549821 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.549562 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2"} err="failed to get container status \"2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2\": rpc error: code = NotFound desc = could not find container \"2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2\": container with ID starting with 2d3f378b86d2b4eeb61d543da35f16849aa23ec307e9a39ad30c1cbb439e36e2 not found: ID does not exist" Sep 15 21:08:21.549821 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.549581 2579 scope.go:117] "RemoveContainer" containerID="e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863" Sep 15 21:08:21.557763 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.557744 2579 scope.go:117] "RemoveContainer" containerID="e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863" Sep 15 21:08:21.558047 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:08:21.558028 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863\": container with ID starting with e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863 not found: ID does not exist" containerID="e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863" Sep 15 21:08:21.558112 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.558052 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863"} err="failed to get container status \"e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863\": rpc error: code = NotFound desc = could not find container \"e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863\": container with ID starting with e4e7d3fd947c0cdcb209ddf7edf2ac2be142e17d50a056d8f0ade904852e6863 not found: ID does not exist" Sep 15 21:08:21.568246 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.568219 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw"] Sep 15 21:08:21.572044 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.571978 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-w0-3f1c93-kserve-6d557c5fd-k6wnw"] Sep 15 21:08:21.586176 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.586145 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c"] Sep 15 21:08:21.588543 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:21.588523 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/embedding-w0-3f1c93-kserve-55f554dccf-2rc6c"] Sep 15 21:08:23.318442 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:23.318412 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199b0526-668c-4746-8455-dcb606fd015b" path="/var/lib/kubelet/pods/199b0526-668c-4746-8455-dcb606fd015b/volumes" Sep 15 21:08:23.318842 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:23.318820 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9edcd1ee-fc26-43df-bce7-d7226a763c25" path="/var/lib/kubelet/pods/9edcd1ee-fc26-43df-bce7-d7226a763c25/volumes" Sep 15 21:08:25.144212 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.144178 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g"] Sep 15 21:08:25.144625 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.144388 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" containerID="cri-o://3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2" gracePeriod=30 Sep 15 21:08:25.234241 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.234195 2579 patch_prober.go:28] interesting pod/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:08:25.234409 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.234267 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:08:25.486350 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486266 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz"] Sep 15 21:08:25.486719 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486704 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9edcd1ee-fc26-43df-bce7-d7226a763c25" containerName="main" Sep 15 21:08:25.486784 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486722 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9edcd1ee-fc26-43df-bce7-d7226a763c25" containerName="main" Sep 15 21:08:25.486784 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486740 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0327ad-a022-4296-86f9-efe56534d05e" containerName="revoke-keys" Sep 15 21:08:25.486784 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486746 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0327ad-a022-4296-86f9-efe56534d05e" containerName="revoke-keys" Sep 15 21:08:25.486784 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486753 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" Sep 15 21:08:25.486784 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486758 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" Sep 15 21:08:25.486784 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486765 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08853ba-5b6c-41bd-8a0f-a519c91cad4a" containerName="payload-processing" Sep 15 21:08:25.486784 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486770 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08853ba-5b6c-41bd-8a0f-a519c91cad4a" containerName="payload-processing" Sep 15 21:08:25.486784 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486783 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01302a9f-6c4a-42d6-812e-adc3897ba19e" containerName="revoke-keys" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486789 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="01302a9f-6c4a-42d6-812e-adc3897ba19e" containerName="revoke-keys" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486799 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486804 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486811 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199b0526-668c-4746-8455-dcb606fd015b" containerName="main" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486816 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="199b0526-668c-4746-8455-dcb606fd015b" containerName="main" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486880 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08853ba-5b6c-41bd-8a0f-a519c91cad4a" containerName="payload-processing" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486888 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f0327ad-a022-4296-86f9-efe56534d05e" containerName="revoke-keys" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486894 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="199b0526-668c-4746-8455-dcb606fd015b" containerName="main" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486899 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486908 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="01302a9f-6c4a-42d6-812e-adc3897ba19e" containerName="revoke-keys" Sep 15 21:08:25.487066 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.486914 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9edcd1ee-fc26-43df-bce7-d7226a763c25" containerName="main" Sep 15 21:08:25.491443 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.491426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" Sep 15 21:08:25.493649 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.493629 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:08:25.499270 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.498955 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz"] Sep 15 21:08:25.597437 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.597403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10615fb2-3250-47a2-844c-e74fd7cb0d97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz\" (UID: \"10615fb2-3250-47a2-844c-e74fd7cb0d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" Sep 15 21:08:25.698580 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.698533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10615fb2-3250-47a2-844c-e74fd7cb0d97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz\" (UID: \"10615fb2-3250-47a2-844c-e74fd7cb0d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" Sep 15 21:08:25.699142 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.699124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10615fb2-3250-47a2-844c-e74fd7cb0d97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz\" (UID: \"10615fb2-3250-47a2-844c-e74fd7cb0d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" Sep 15 21:08:25.804079 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:25.804042 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" Sep 15 21:08:26.078414 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:26.078386 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz"] Sep 15 21:08:26.080693 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:08:26.080666 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10615fb2_3250_47a2_844c_e74fd7cb0d97.slice/crio-06d455f44a6b6c8e994df61c9be33c1fc478c9f3355cc6cb8a82a2abdb097d47 WatchSource:0}: Error finding container 06d455f44a6b6c8e994df61c9be33c1fc478c9f3355cc6cb8a82a2abdb097d47: Status 404 returned error can't find the container with id 06d455f44a6b6c8e994df61c9be33c1fc478c9f3355cc6cb8a82a2abdb097d47 Sep 15 21:08:26.574475 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:26.574442 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" event={"ID":"10615fb2-3250-47a2-844c-e74fd7cb0d97","Type":"ContainerStarted","Data":"8b3f1a230b6b85e44d34c5c11cbaf0f70ee79430cb1cbec43df5bb5d1ddb10cc"} Sep 15 21:08:26.574880 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:26.574489 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" event={"ID":"10615fb2-3250-47a2-844c-e74fd7cb0d97","Type":"ContainerStarted","Data":"06d455f44a6b6c8e994df61c9be33c1fc478c9f3355cc6cb8a82a2abdb097d47"} Sep 15 21:08:26.595624 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:26.595572 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" podStartSLOduration=1.595553979 podStartE2EDuration="1.595553979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:26.594575391 +0000 UTC m=+1903.788528846" watchObservedRunningTime="2026-09-15 21:08:26.595553979 +0000 UTC m=+1903.789507433" Sep 15 21:08:27.581829 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:27.581790 2579 generic.go:356] "Generic (PLEG): container finished" podID="10615fb2-3250-47a2-844c-e74fd7cb0d97" containerID="8b3f1a230b6b85e44d34c5c11cbaf0f70ee79430cb1cbec43df5bb5d1ddb10cc" exitCode=0 Sep 15 21:08:27.582279 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:27.581878 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" event={"ID":"10615fb2-3250-47a2-844c-e74fd7cb0d97","Type":"ContainerDied","Data":"8b3f1a230b6b85e44d34c5c11cbaf0f70ee79430cb1cbec43df5bb5d1ddb10cc"} Sep 15 21:08:28.724225 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:28.724199 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" Sep 15 21:08:28.829802 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:28.829768 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10615fb2-3250-47a2-844c-e74fd7cb0d97-maas-api-service-ca\") pod \"10615fb2-3250-47a2-844c-e74fd7cb0d97\" (UID: \"10615fb2-3250-47a2-844c-e74fd7cb0d97\") " Sep 15 21:08:28.830099 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:28.830073 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10615fb2-3250-47a2-844c-e74fd7cb0d97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10615fb2-3250-47a2-844c-e74fd7cb0d97" (UID: "10615fb2-3250-47a2-844c-e74fd7cb0d97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:28.931168 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:28.931084 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10615fb2-3250-47a2-844c-e74fd7cb0d97-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:29.595185 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:29.595102 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" Sep 15 21:08:29.595185 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:29.595134 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz" event={"ID":"10615fb2-3250-47a2-844c-e74fd7cb0d97","Type":"ContainerDied","Data":"06d455f44a6b6c8e994df61c9be33c1fc478c9f3355cc6cb8a82a2abdb097d47"} Sep 15 21:08:29.595185 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:29.595182 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06d455f44a6b6c8e994df61c9be33c1fc478c9f3355cc6cb8a82a2abdb097d47" Sep 15 21:08:30.396306 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.396277 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:08:30.549987 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.549958 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-data\") pod \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " Sep 15 21:08:30.550181 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550005 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-podinfo\") pod \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " Sep 15 21:08:30.550181 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550029 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-certs\") pod \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " Sep 15 21:08:30.550181 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550057 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-envoy\") pod \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " Sep 15 21:08:30.550181 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550088 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65pp5\" (UniqueName: \"kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-kube-api-access-65pp5\") pod \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " Sep 15 21:08:30.550181 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550118 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-socket\") pod \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " Sep 15 21:08:30.550181 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550161 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-credential-socket\") pod \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " Sep 15 21:08:30.550490 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550209 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-token\") pod \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " Sep 15 21:08:30.550490 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550215 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" (UID: "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:30.550490 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550321 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istiod-ca-cert\") pod \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\" (UID: \"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2\") " Sep 15 21:08:30.550490 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550419 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" (UID: "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:30.550490 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550436 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" (UID: "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:30.550964 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550607 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" (UID: "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:30.550964 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550745 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" (UID: "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:30.550964 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550761 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:30.550964 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550953 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:30.551195 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550967 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:30.551195 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.550981 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:30.552483 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.552452 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-kube-api-access-65pp5" (OuterVolumeSpecName: "kube-api-access-65pp5") pod "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" (UID: "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2"). InnerVolumeSpecName "kube-api-access-65pp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:30.552660 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.552525 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" (UID: "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:08:30.552660 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.552529 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" (UID: "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:30.552660 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.552587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" (UID: "cb5ae23d-ac4a-4969-8c0d-e8f788e865f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:30.603090 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.603059 2579 generic.go:356] "Generic (PLEG): container finished" podID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerID="3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2" exitCode=0 Sep 15 21:08:30.603261 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.603109 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" Sep 15 21:08:30.603261 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.603144 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" event={"ID":"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2","Type":"ContainerDied","Data":"3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2"} Sep 15 21:08:30.603261 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.603203 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g" event={"ID":"cb5ae23d-ac4a-4969-8c0d-e8f788e865f2","Type":"ContainerDied","Data":"81204a932ef1565f093a12422f46391c8ccffa690af88442f603da99279e8c2c"} Sep 15 21:08:30.603261 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.603226 2579 scope.go:117] "RemoveContainer" containerID="3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2" Sep 15 21:08:30.613524 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.613489 2579 scope.go:117] "RemoveContainer" containerID="9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6" Sep 15 21:08:30.622078 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.622052 2579 scope.go:117] "RemoveContainer" containerID="3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2" Sep 15 21:08:30.622490 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:08:30.622461 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2\": container with ID starting with 3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2 not found: ID does not exist" containerID="3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2" Sep 15 21:08:30.622639 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.622522 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2"} err="failed to get container status \"3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2\": rpc error: code = NotFound desc = could not find container \"3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2\": container with ID starting with 3a04c42ffd070a1fb92762d3202a739cb5cb1a620493f2108c9af8cb2cea3fd2 not found: ID does not exist" Sep 15 21:08:30.622639 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.622549 2579 scope.go:117] "RemoveContainer" containerID="9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6" Sep 15 21:08:30.622906 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:08:30.622885 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6\": container with ID starting with 9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6 not found: ID does not exist" containerID="9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6" Sep 15 21:08:30.622970 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.622911 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6"} err="failed to get container status \"9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6\": rpc error: code = NotFound desc = could not find container \"9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6\": container with ID starting with 9166ae16fd6455197596b5f54abe888da3c1542cd1f63c524a0f92113a0c73f6 not found: ID does not exist" Sep 15 21:08:30.625095 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.625071 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g"] Sep 15 21:08:30.628752 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.628730 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-14aeb7-openshift-default-8fdf87c58-t9j2g"] Sep 15 21:08:30.652100 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.652072 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:30.652100 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.652100 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:30.652253 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.652115 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65pp5\" (UniqueName: \"kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-kube-api-access-65pp5\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:30.652253 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.652128 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:30.652253 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:30.652139 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:31.314674 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:31.314643 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" path="/var/lib/kubelet/pods/cb5ae23d-ac4a-4969-8c0d-e8f788e865f2/volumes" Sep 15 21:08:35.955313 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:35.955274 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb"] Sep 15 21:08:35.959553 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:35.959521 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-de4215-29825100-hxxsb"] Sep 15 21:08:36.150229 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:36.150187 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz"] Sep 15 21:08:36.153542 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:36.153513 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de4215-xw4dz"] Sep 15 21:08:37.312654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:37.312622 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10615fb2-3250-47a2-844c-e74fd7cb0d97" path="/var/lib/kubelet/pods/10615fb2-3250-47a2-844c-e74fd7cb0d97/volumes" Sep 15 21:08:37.313028 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:37.312934 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81436216-f5ed-4497-9727-525f1c1db457" path="/var/lib/kubelet/pods/81436216-f5ed-4497-9727-525f1c1db457/volumes" Sep 15 21:08:43.706985 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:43.706948 2579 scope.go:117] "RemoveContainer" containerID="6bf0a5c3540277447422f7b7b5a2c4e27bb7724cabccff8d2dae2d3a21d5f94f" Sep 15 21:08:43.717932 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:43.717903 2579 scope.go:117] "RemoveContainer" containerID="7d5e7d6d149821732ddcfb0948097710d176bff2e041136154f89c2fa1cad85f" Sep 15 21:08:44.080699 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.080660 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8"] Sep 15 21:08:44.082114 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.082063 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" Sep 15 21:08:44.082609 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.082329 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" Sep 15 21:08:44.082609 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.082399 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10615fb2-3250-47a2-844c-e74fd7cb0d97" containerName="revoke-keys" Sep 15 21:08:44.082609 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.082409 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="10615fb2-3250-47a2-844c-e74fd7cb0d97" containerName="revoke-keys" Sep 15 21:08:44.082609 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.082446 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" Sep 15 21:08:44.082609 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.082455 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" Sep 15 21:08:44.083158 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.083122 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a02c7a5-84eb-4469-b758-1e238310af13" containerName="istio-proxy" Sep 15 21:08:44.083316 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.083266 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" Sep 15 21:08:44.083316 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.083286 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="10615fb2-3250-47a2-844c-e74fd7cb0d97" containerName="revoke-keys" Sep 15 21:08:44.089408 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.089381 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8" Sep 15 21:08:44.092051 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.092032 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:08:44.092280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.092255 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8"] Sep 15 21:08:44.190003 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.189965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bad8fd-4b56-4b92-b6a3-218cd0248b49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8\" (UID: \"70bad8fd-4b56-4b92-b6a3-218cd0248b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8" Sep 15 21:08:44.291462 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.291423 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bad8fd-4b56-4b92-b6a3-218cd0248b49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8\" (UID: \"70bad8fd-4b56-4b92-b6a3-218cd0248b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8" Sep 15 21:08:44.292045 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.292024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bad8fd-4b56-4b92-b6a3-218cd0248b49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8\" (UID: \"70bad8fd-4b56-4b92-b6a3-218cd0248b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8" Sep 15 21:08:44.403081 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.402994 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8" Sep 15 21:08:44.682722 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:44.682697 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8"] Sep 15 21:08:44.684891 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:08:44.684866 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bad8fd_4b56_4b92_b6a3_218cd0248b49.slice/crio-4f8352837f9a227bdb97755b491eb2b27f9faaccd5d07ae8de91ae2d2b3b0434 WatchSource:0}: Error finding container 4f8352837f9a227bdb97755b491eb2b27f9faaccd5d07ae8de91ae2d2b3b0434: Status 404 returned error can't find the container with id 4f8352837f9a227bdb97755b491eb2b27f9faaccd5d07ae8de91ae2d2b3b0434 Sep 15 21:08:45.688672 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:45.688584 2579 generic.go:356] "Generic (PLEG): container finished" podID="70bad8fd-4b56-4b92-b6a3-218cd0248b49" containerID="1da090bf17f1e18db7a6cd2439e91dd28a7da9d02ff1a46d9bc65c6335ec8e17" exitCode=0 Sep 15 21:08:45.688672 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:45.688639 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8" event={"ID":"70bad8fd-4b56-4b92-b6a3-218cd0248b49","Type":"ContainerDied","Data":"1da090bf17f1e18db7a6cd2439e91dd28a7da9d02ff1a46d9bc65c6335ec8e17"} Sep 15 21:08:45.689107 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:45.688681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8" event={"ID":"70bad8fd-4b56-4b92-b6a3-218cd0248b49","Type":"ContainerStarted","Data":"4f8352837f9a227bdb97755b491eb2b27f9faaccd5d07ae8de91ae2d2b3b0434"} Sep 15 21:08:46.833975 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:46.833948 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8" Sep 15 21:08:46.914591 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:46.914561 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bad8fd-4b56-4b92-b6a3-218cd0248b49-maas-api-service-ca\") pod \"70bad8fd-4b56-4b92-b6a3-218cd0248b49\" (UID: \"70bad8fd-4b56-4b92-b6a3-218cd0248b49\") " Sep 15 21:08:46.914929 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:46.914905 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bad8fd-4b56-4b92-b6a3-218cd0248b49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70bad8fd-4b56-4b92-b6a3-218cd0248b49" (UID: "70bad8fd-4b56-4b92-b6a3-218cd0248b49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:47.015410 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:47.015380 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bad8fd-4b56-4b92-b6a3-218cd0248b49-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:47.701116 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:47.701034 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8" Sep 15 21:08:47.701116 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:47.701048 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8" event={"ID":"70bad8fd-4b56-4b92-b6a3-218cd0248b49","Type":"ContainerDied","Data":"4f8352837f9a227bdb97755b491eb2b27f9faaccd5d07ae8de91ae2d2b3b0434"} Sep 15 21:08:47.701116 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:47.701093 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f8352837f9a227bdb97755b491eb2b27f9faaccd5d07ae8de91ae2d2b3b0434" Sep 15 21:08:54.618563 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.618529 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh"] Sep 15 21:08:54.618995 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.618752 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" podUID="afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f" containerName="payload-processing" containerID="cri-o://e87a97a655bd0d9b0a903f49c29bcef1f799c736584b2bd96adaba7beaf5d7cd" gracePeriod=30 Sep 15 21:08:54.741146 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.741114 2579 generic.go:356] "Generic (PLEG): container finished" podID="afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f" containerID="e87a97a655bd0d9b0a903f49c29bcef1f799c736584b2bd96adaba7beaf5d7cd" exitCode=0 Sep 15 21:08:54.741275 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.741195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" event={"ID":"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f","Type":"ContainerDied","Data":"e87a97a655bd0d9b0a903f49c29bcef1f799c736584b2bd96adaba7beaf5d7cd"} Sep 15 21:08:54.792375 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.792344 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8"] Sep 15 21:08:54.800932 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.796124 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3825fc-f28m8"] Sep 15 21:08:54.891155 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.891084 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:08:54.994583 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.994552 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-plugins-config-volume\") pod \"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f\" (UID: \"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f\") " Sep 15 21:08:54.994757 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.994662 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d5xq\" (UniqueName: \"kubernetes.io/projected/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-kube-api-access-9d5xq\") pod \"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f\" (UID: \"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f\") " Sep 15 21:08:54.995031 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.995004 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f" (UID: "afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:54.996872 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:54.996846 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-kube-api-access-9d5xq" (OuterVolumeSpecName: "kube-api-access-9d5xq") pod "afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f" (UID: "afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f"). InnerVolumeSpecName "kube-api-access-9d5xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:55.096267 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:55.096226 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d5xq\" (UniqueName: \"kubernetes.io/projected/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-kube-api-access-9d5xq\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:55.096267 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:55.096262 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:08:55.311688 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:55.311654 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70bad8fd-4b56-4b92-b6a3-218cd0248b49" path="/var/lib/kubelet/pods/70bad8fd-4b56-4b92-b6a3-218cd0248b49/volumes" Sep 15 21:08:55.748182 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:55.748098 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" Sep 15 21:08:55.748182 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:55.748119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh" event={"ID":"afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f","Type":"ContainerDied","Data":"a6bc3994e858cf50f66e3b1bddcddd93fc6ca7c868d1632e6ac20bcefbb871f3"} Sep 15 21:08:55.748182 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:55.748174 2579 scope.go:117] "RemoveContainer" containerID="e87a97a655bd0d9b0a903f49c29bcef1f799c736584b2bd96adaba7beaf5d7cd" Sep 15 21:08:55.783177 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:55.783150 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh"] Sep 15 21:08:55.787835 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:55.787803 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3825fc-5c65db9f6f-8swqh"] Sep 15 21:08:57.311556 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:08:57.311524 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f" path="/var/lib/kubelet/pods/afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f/volumes" Sep 15 21:09:28.896031 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:28.895987 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp"] Sep 15 21:09:29.794205 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.794170 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd"] Sep 15 21:09:29.794597 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.794583 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f" containerName="payload-processing" Sep 15 21:09:29.794662 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.794599 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f" containerName="payload-processing" Sep 15 21:09:29.794662 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.794612 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bad8fd-4b56-4b92-b6a3-218cd0248b49" containerName="revoke-keys" Sep 15 21:09:29.794662 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.794617 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bad8fd-4b56-4b92-b6a3-218cd0248b49" containerName="revoke-keys" Sep 15 21:09:29.794779 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.794678 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5ae23d-ac4a-4969-8c0d-e8f788e865f2" containerName="istio-proxy" Sep 15 21:09:29.794779 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.794690 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70bad8fd-4b56-4b92-b6a3-218cd0248b49" containerName="revoke-keys" Sep 15 21:09:29.794779 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.794699 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="afcaabf4-f53b-4fbb-bbc3-0a5c72714f4f" containerName="payload-processing" Sep 15 21:09:29.797981 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.797962 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" Sep 15 21:09:29.800417 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.800394 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:09:29.804753 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.804724 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd"] Sep 15 21:09:29.840436 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.840397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22225a81-58b1-4598-81ee-0c49eff2b0ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd\" (UID: \"22225a81-58b1-4598-81ee-0c49eff2b0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" Sep 15 21:09:29.941262 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.941227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22225a81-58b1-4598-81ee-0c49eff2b0ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd\" (UID: \"22225a81-58b1-4598-81ee-0c49eff2b0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" Sep 15 21:09:29.942089 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:29.942065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22225a81-58b1-4598-81ee-0c49eff2b0ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd\" (UID: \"22225a81-58b1-4598-81ee-0c49eff2b0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" Sep 15 21:09:30.111051 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:30.110962 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" Sep 15 21:09:30.382962 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:30.382938 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd"] Sep 15 21:09:30.385031 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:09:30.384996 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22225a81_58b1_4598_81ee_0c49eff2b0ea.slice/crio-fcffb683f3bbd5b2a077611b05df20a8a45841f73420aa07782e0f60c781ccbd WatchSource:0}: Error finding container fcffb683f3bbd5b2a077611b05df20a8a45841f73420aa07782e0f60c781ccbd: Status 404 returned error can't find the container with id fcffb683f3bbd5b2a077611b05df20a8a45841f73420aa07782e0f60c781ccbd Sep 15 21:09:30.933797 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:30.933708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" event={"ID":"22225a81-58b1-4598-81ee-0c49eff2b0ea","Type":"ContainerStarted","Data":"d63f4f653c6fe2231e48ab9a9b67c0a317c14d26a212c6bf0ff06a6d182b269b"} Sep 15 21:09:30.933797 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:30.933766 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" event={"ID":"22225a81-58b1-4598-81ee-0c49eff2b0ea","Type":"ContainerStarted","Data":"fcffb683f3bbd5b2a077611b05df20a8a45841f73420aa07782e0f60c781ccbd"} Sep 15 21:09:30.948927 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:30.948858 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" podStartSLOduration=1.9488407570000001 podStartE2EDuration="1.948840757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:30.948429177 +0000 UTC m=+1968.142382632" watchObservedRunningTime="2026-09-15 21:09:30.948840757 +0000 UTC m=+1968.142794213" Sep 15 21:09:31.940412 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:31.940373 2579 generic.go:356] "Generic (PLEG): container finished" podID="22225a81-58b1-4598-81ee-0c49eff2b0ea" containerID="d63f4f653c6fe2231e48ab9a9b67c0a317c14d26a212c6bf0ff06a6d182b269b" exitCode=0 Sep 15 21:09:31.940633 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:31.940462 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" event={"ID":"22225a81-58b1-4598-81ee-0c49eff2b0ea","Type":"ContainerDied","Data":"d63f4f653c6fe2231e48ab9a9b67c0a317c14d26a212c6bf0ff06a6d182b269b"} Sep 15 21:09:33.083625 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:33.083603 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" Sep 15 21:09:33.169977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:33.169933 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22225a81-58b1-4598-81ee-0c49eff2b0ea-maas-api-service-ca\") pod \"22225a81-58b1-4598-81ee-0c49eff2b0ea\" (UID: \"22225a81-58b1-4598-81ee-0c49eff2b0ea\") " Sep 15 21:09:33.170345 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:33.170320 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22225a81-58b1-4598-81ee-0c49eff2b0ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22225a81-58b1-4598-81ee-0c49eff2b0ea" (UID: "22225a81-58b1-4598-81ee-0c49eff2b0ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:09:33.270991 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:33.270946 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22225a81-58b1-4598-81ee-0c49eff2b0ea-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:33.953256 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:33.953227 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" Sep 15 21:09:33.953448 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:33.953267 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd" event={"ID":"22225a81-58b1-4598-81ee-0c49eff2b0ea","Type":"ContainerDied","Data":"fcffb683f3bbd5b2a077611b05df20a8a45841f73420aa07782e0f60c781ccbd"} Sep 15 21:09:33.953448 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:33.953322 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcffb683f3bbd5b2a077611b05df20a8a45841f73420aa07782e0f60c781ccbd" Sep 15 21:09:34.650297 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:34.650213 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz"] Sep 15 21:09:34.655663 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:34.655632 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr"] Sep 15 21:09:34.664211 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:34.664179 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n"] Sep 15 21:09:34.676207 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:34.676167 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw"] Sep 15 21:09:43.916437 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:43.916370 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" podUID="5bfd3d03-d232-4d83-a394-7e75e3db412d" containerName="main" containerID="cri-o://52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c" gracePeriod=45 Sep 15 21:09:44.170137 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.170106 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:09:44.276609 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.276572 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-home\") pod \"5bfd3d03-d232-4d83-a394-7e75e3db412d\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " Sep 15 21:09:44.276810 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.276665 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-tmp-dir\") pod \"5bfd3d03-d232-4d83-a394-7e75e3db412d\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " Sep 15 21:09:44.276810 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.276712 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-dshm\") pod \"5bfd3d03-d232-4d83-a394-7e75e3db412d\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " Sep 15 21:09:44.276810 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.276743 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-model-cache\") pod \"5bfd3d03-d232-4d83-a394-7e75e3db412d\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " Sep 15 21:09:44.276986 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.276843 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cx6n\" (UniqueName: \"kubernetes.io/projected/5bfd3d03-d232-4d83-a394-7e75e3db412d-kube-api-access-5cx6n\") pod \"5bfd3d03-d232-4d83-a394-7e75e3db412d\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " Sep 15 21:09:44.276986 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.276879 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfd3d03-d232-4d83-a394-7e75e3db412d-tls-certs\") pod \"5bfd3d03-d232-4d83-a394-7e75e3db412d\" (UID: \"5bfd3d03-d232-4d83-a394-7e75e3db412d\") " Sep 15 21:09:44.276986 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.276885 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-home" (OuterVolumeSpecName: "home") pod "5bfd3d03-d232-4d83-a394-7e75e3db412d" (UID: "5bfd3d03-d232-4d83-a394-7e75e3db412d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:44.276986 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.276949 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bfd3d03-d232-4d83-a394-7e75e3db412d" (UID: "5bfd3d03-d232-4d83-a394-7e75e3db412d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:44.277209 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.277185 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bfd3d03-d232-4d83-a394-7e75e3db412d" (UID: "5bfd3d03-d232-4d83-a394-7e75e3db412d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:44.277360 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.277212 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:44.277360 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.277239 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:44.279311 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.279286 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfd3d03-d232-4d83-a394-7e75e3db412d-kube-api-access-5cx6n" (OuterVolumeSpecName: "kube-api-access-5cx6n") pod "5bfd3d03-d232-4d83-a394-7e75e3db412d" (UID: "5bfd3d03-d232-4d83-a394-7e75e3db412d"). InnerVolumeSpecName "kube-api-access-5cx6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:44.279311 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.279301 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-dshm" (OuterVolumeSpecName: "dshm") pod "5bfd3d03-d232-4d83-a394-7e75e3db412d" (UID: "5bfd3d03-d232-4d83-a394-7e75e3db412d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:44.279454 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.279315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfd3d03-d232-4d83-a394-7e75e3db412d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bfd3d03-d232-4d83-a394-7e75e3db412d" (UID: "5bfd3d03-d232-4d83-a394-7e75e3db412d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:09:44.378483 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.378428 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:44.378483 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.378476 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bfd3d03-d232-4d83-a394-7e75e3db412d-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:44.378483 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.378491 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cx6n\" (UniqueName: \"kubernetes.io/projected/5bfd3d03-d232-4d83-a394-7e75e3db412d-kube-api-access-5cx6n\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:44.378483 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:44.378529 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfd3d03-d232-4d83-a394-7e75e3db412d-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:45.017801 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.017764 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bfd3d03-d232-4d83-a394-7e75e3db412d" containerID="52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c" exitCode=0 Sep 15 21:09:45.018224 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.017809 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" Sep 15 21:09:45.018224 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.017848 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" event={"ID":"5bfd3d03-d232-4d83-a394-7e75e3db412d","Type":"ContainerDied","Data":"52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c"} Sep 15 21:09:45.018224 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.017895 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp" event={"ID":"5bfd3d03-d232-4d83-a394-7e75e3db412d","Type":"ContainerDied","Data":"872391a3a588fd3d49f50c73a3f3e73496579c124ca70af9a127fa1b845393b1"} Sep 15 21:09:45.018224 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.017917 2579 scope.go:117] "RemoveContainer" containerID="52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c" Sep 15 21:09:45.027894 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.027875 2579 scope.go:117] "RemoveContainer" containerID="52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c" Sep 15 21:09:45.028209 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:09:45.028189 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c\": container with ID starting with 52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c not found: ID does not exist" containerID="52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c" Sep 15 21:09:45.028295 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.028218 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c"} err="failed to get container status \"52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c\": rpc error: code = NotFound desc = could not find container \"52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c\": container with ID starting with 52b03cbe40d10bccb6d7ac981342a162280d501bacddda06b7249036299f716c not found: ID does not exist" Sep 15 21:09:45.038307 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.038277 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp"] Sep 15 21:09:45.046046 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.046023 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/auth-test-model-280fb4-kserve-77df8dc6c-45ccp"] Sep 15 21:09:45.316234 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.316148 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bfd3d03-d232-4d83-a394-7e75e3db412d" path="/var/lib/kubelet/pods/5bfd3d03-d232-4d83-a394-7e75e3db412d/volumes" Sep 15 21:09:45.422107 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.422075 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf"] Sep 15 21:09:45.422369 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.422344 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" podUID="1ce77d63-7610-47c6-ae43-64b4eb7da4bd" containerName="payload-processing" containerID="cri-o://67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3" gracePeriod=30 Sep 15 21:09:45.593749 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.593663 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd"] Sep 15 21:09:45.598784 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.598755 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-16bde4-84trd"] Sep 15 21:09:45.686140 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.686116 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 21:09:45.792901 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.792868 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-plugins-config-volume\") pod \"1ce77d63-7610-47c6-ae43-64b4eb7da4bd\" (UID: \"1ce77d63-7610-47c6-ae43-64b4eb7da4bd\") " Sep 15 21:09:45.792901 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.792914 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-488wc\" (UniqueName: \"kubernetes.io/projected/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-kube-api-access-488wc\") pod \"1ce77d63-7610-47c6-ae43-64b4eb7da4bd\" (UID: \"1ce77d63-7610-47c6-ae43-64b4eb7da4bd\") " Sep 15 21:09:45.793286 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.793261 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ce77d63-7610-47c6-ae43-64b4eb7da4bd" (UID: "1ce77d63-7610-47c6-ae43-64b4eb7da4bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:09:45.795185 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.795161 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-kube-api-access-488wc" (OuterVolumeSpecName: "kube-api-access-488wc") pod "1ce77d63-7610-47c6-ae43-64b4eb7da4bd" (UID: "1ce77d63-7610-47c6-ae43-64b4eb7da4bd"). InnerVolumeSpecName "kube-api-access-488wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:45.894684 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.894597 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-488wc\" (UniqueName: \"kubernetes.io/projected/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-kube-api-access-488wc\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:45.894684 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:45.894624 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce77d63-7610-47c6-ae43-64b4eb7da4bd-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:46.024289 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:46.024255 2579 generic.go:356] "Generic (PLEG): container finished" podID="1ce77d63-7610-47c6-ae43-64b4eb7da4bd" containerID="67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3" exitCode=0 Sep 15 21:09:46.024829 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:46.024303 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" Sep 15 21:09:46.024829 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:46.024347 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" event={"ID":"1ce77d63-7610-47c6-ae43-64b4eb7da4bd","Type":"ContainerDied","Data":"67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3"} Sep 15 21:09:46.024829 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:46.024407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf" event={"ID":"1ce77d63-7610-47c6-ae43-64b4eb7da4bd","Type":"ContainerDied","Data":"37135893602d7c9865f8e1e46a2ce818c69db1eb479528404bc2c231b1ff9e94"} Sep 15 21:09:46.024829 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:46.024440 2579 scope.go:117] "RemoveContainer" containerID="67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3" Sep 15 21:09:46.034654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:46.034627 2579 scope.go:117] "RemoveContainer" containerID="67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3" Sep 15 21:09:46.034954 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:09:46.034936 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3\": container with ID starting with 67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3 not found: ID does not exist" containerID="67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3" Sep 15 21:09:46.035003 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:46.034961 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3"} err="failed to get container status \"67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3\": rpc error: code = NotFound desc = could not find container \"67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3\": container with ID starting with 67e0e0e81faae49176bd969cce9f2674a2a60514f43bb8dd0968bbfcaa4dd8d3 not found: ID does not exist" Sep 15 21:09:46.045255 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:46.045232 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf"] Sep 15 21:09:46.051630 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:46.051609 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-16bde4-565dd6d69f-nc2kf"] Sep 15 21:09:47.311767 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:47.311734 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ce77d63-7610-47c6-ae43-64b4eb7da4bd" path="/var/lib/kubelet/pods/1ce77d63-7610-47c6-ae43-64b4eb7da4bd/volumes" Sep 15 21:09:47.312177 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:47.312071 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22225a81-58b1-4598-81ee-0c49eff2b0ea" path="/var/lib/kubelet/pods/22225a81-58b1-4598-81ee-0c49eff2b0ea/volumes" Sep 15 21:09:49.671414 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:49.671368 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" podUID="27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" containerName="main" containerID="cri-o://76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f" gracePeriod=45 Sep 15 21:09:49.678061 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:49.678024 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" podUID="03557738-ca55-4c4d-aa1a-ef659efe0f40" containerName="main" containerID="cri-o://2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9" gracePeriod=45 Sep 15 21:09:49.684158 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:49.684088 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" podUID="3b174a8a-fe9f-4360-b931-0c9016c8832c" containerName="main" containerID="cri-o://552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea" gracePeriod=45 Sep 15 21:09:49.695372 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:49.695341 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" podUID="9afe40ed-514d-4fd3-a64b-4c48549890a3" containerName="main" containerID="cri-o://b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84" gracePeriod=45 Sep 15 21:09:49.972155 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:49.972128 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 21:09:49.990025 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:49.989986 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:09:50.005764 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.005736 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:09:50.016616 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.016580 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 21:09:50.052706 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.052670 2579 generic.go:356] "Generic (PLEG): container finished" podID="03557738-ca55-4c4d-aa1a-ef659efe0f40" containerID="2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9" exitCode=0 Sep 15 21:09:50.052908 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.052759 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" Sep 15 21:09:50.052908 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.052800 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" event={"ID":"03557738-ca55-4c4d-aa1a-ef659efe0f40","Type":"ContainerDied","Data":"2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9"} Sep 15 21:09:50.052908 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.052841 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr" event={"ID":"03557738-ca55-4c4d-aa1a-ef659efe0f40","Type":"ContainerDied","Data":"50213034a27da97392ee2801ca66967134de8f1febccb5bdadc96f810e89a420"} Sep 15 21:09:50.052908 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.052865 2579 scope.go:117] "RemoveContainer" containerID="2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9" Sep 15 21:09:50.054517 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.054455 2579 generic.go:356] "Generic (PLEG): container finished" podID="3b174a8a-fe9f-4360-b931-0c9016c8832c" containerID="552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea" exitCode=0 Sep 15 21:09:50.054650 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.054514 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" Sep 15 21:09:50.054650 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.054559 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" event={"ID":"3b174a8a-fe9f-4360-b931-0c9016c8832c","Type":"ContainerDied","Data":"552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea"} Sep 15 21:09:50.054650 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.054618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n" event={"ID":"3b174a8a-fe9f-4360-b931-0c9016c8832c","Type":"ContainerDied","Data":"fbdf4d988d5b3a6227e7ee9f9d268b4231263dc2865a87069ac449edc3d1fbcd"} Sep 15 21:09:50.056671 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.056225 2579 generic.go:356] "Generic (PLEG): container finished" podID="27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" containerID="76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f" exitCode=0 Sep 15 21:09:50.056671 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.056258 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" event={"ID":"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0","Type":"ContainerDied","Data":"76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f"} Sep 15 21:09:50.056671 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.056292 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" event={"ID":"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0","Type":"ContainerDied","Data":"938dbda31e495479bd23cd16da41da5f4a95c0fd2d44beea95a65ed1aa49b661"} Sep 15 21:09:50.056671 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.056347 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz" Sep 15 21:09:50.058100 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.058077 2579 generic.go:356] "Generic (PLEG): container finished" podID="9afe40ed-514d-4fd3-a64b-4c48549890a3" containerID="b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84" exitCode=0 Sep 15 21:09:50.058208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.058117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" event={"ID":"9afe40ed-514d-4fd3-a64b-4c48549890a3","Type":"ContainerDied","Data":"b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84"} Sep 15 21:09:50.058208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.058144 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" event={"ID":"9afe40ed-514d-4fd3-a64b-4c48549890a3","Type":"ContainerDied","Data":"cfbbadd23462c16e23695d2a9aa4ab2cd9bc98729d281a7f5b1d35cb9b5eef8b"} Sep 15 21:09:50.058208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.058120 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw" Sep 15 21:09:50.064937 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.064915 2579 scope.go:117] "RemoveContainer" containerID="2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9" Sep 15 21:09:50.065229 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:09:50.065209 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9\": container with ID starting with 2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9 not found: ID does not exist" containerID="2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9" Sep 15 21:09:50.065326 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.065241 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9"} err="failed to get container status \"2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9\": rpc error: code = NotFound desc = could not find container \"2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9\": container with ID starting with 2c9db4a32dd4749003dc8a899bc0b79903981bff0818b5341881e89bccba2fb9 not found: ID does not exist" Sep 15 21:09:50.065326 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.065258 2579 scope.go:117] "RemoveContainer" containerID="552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea" Sep 15 21:09:50.073870 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.073848 2579 scope.go:117] "RemoveContainer" containerID="552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea" Sep 15 21:09:50.074179 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:09:50.074161 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea\": container with ID starting with 552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea not found: ID does not exist" containerID="552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea" Sep 15 21:09:50.074244 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.074186 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea"} err="failed to get container status \"552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea\": rpc error: code = NotFound desc = could not find container \"552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea\": container with ID starting with 552f32211da184c33415beb71bcae27c6150cb76044b97dbd6ac10bdeee79fea not found: ID does not exist" Sep 15 21:09:50.074244 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.074203 2579 scope.go:117] "RemoveContainer" containerID="76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f" Sep 15 21:09:50.085776 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.085754 2579 scope.go:117] "RemoveContainer" containerID="76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f" Sep 15 21:09:50.086146 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:09:50.086118 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f\": container with ID starting with 76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f not found: ID does not exist" containerID="76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f" Sep 15 21:09:50.086208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.086157 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f"} err="failed to get container status \"76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f\": rpc error: code = NotFound desc = could not find container \"76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f\": container with ID starting with 76b39aa96eb68c74f962d8706243bf813dbc632c591ebb071068c73a1d83307f not found: ID does not exist" Sep 15 21:09:50.086208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.086181 2579 scope.go:117] "RemoveContainer" containerID="b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84" Sep 15 21:09:50.095854 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.095832 2579 scope.go:117] "RemoveContainer" containerID="b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84" Sep 15 21:09:50.096139 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:09:50.096117 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84\": container with ID starting with b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84 not found: ID does not exist" containerID="b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84" Sep 15 21:09:50.096181 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.096148 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84"} err="failed to get container status \"b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84\": rpc error: code = NotFound desc = could not find container \"b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84\": container with ID starting with b1eb39898f2362dbddd5eb585df14a3f22c289a0ff70bad12e16f28367ecee84 not found: ID does not exist" Sep 15 21:09:50.136484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136455 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tls-certs\") pod \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " Sep 15 21:09:50.136644 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136490 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-home\") pod \"03557738-ca55-4c4d-aa1a-ef659efe0f40\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " Sep 15 21:09:50.136644 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136547 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq6mz\" (UniqueName: \"kubernetes.io/projected/03557738-ca55-4c4d-aa1a-ef659efe0f40-kube-api-access-sq6mz\") pod \"03557738-ca55-4c4d-aa1a-ef659efe0f40\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " Sep 15 21:09:50.136644 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136572 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b174a8a-fe9f-4360-b931-0c9016c8832c-tls-certs\") pod \"3b174a8a-fe9f-4360-b931-0c9016c8832c\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " Sep 15 21:09:50.136644 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136603 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-tmp-dir\") pod \"9afe40ed-514d-4fd3-a64b-4c48549890a3\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " Sep 15 21:09:50.136644 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136621 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-tmp-dir\") pod \"03557738-ca55-4c4d-aa1a-ef659efe0f40\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " Sep 15 21:09:50.136644 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136645 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afe40ed-514d-4fd3-a64b-4c48549890a3-tls-certs\") pod \"9afe40ed-514d-4fd3-a64b-4c48549890a3\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136675 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-dshm\") pod \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136712 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-model-cache\") pod \"03557738-ca55-4c4d-aa1a-ef659efe0f40\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136756 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-home\") pod \"3b174a8a-fe9f-4360-b931-0c9016c8832c\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tmp-dir\") pod \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136815 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd9g7\" (UniqueName: \"kubernetes.io/projected/3b174a8a-fe9f-4360-b931-0c9016c8832c-kube-api-access-hd9g7\") pod \"3b174a8a-fe9f-4360-b931-0c9016c8832c\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136842 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-home" (OuterVolumeSpecName: "home") pod "03557738-ca55-4c4d-aa1a-ef659efe0f40" (UID: "03557738-ca55-4c4d-aa1a-ef659efe0f40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136853 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-dshm\") pod \"03557738-ca55-4c4d-aa1a-ef659efe0f40\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136869 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9afe40ed-514d-4fd3-a64b-4c48549890a3" (UID: "9afe40ed-514d-4fd3-a64b-4c48549890a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136879 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03557738-ca55-4c4d-aa1a-ef659efe0f40" (UID: "03557738-ca55-4c4d-aa1a-ef659efe0f40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136930 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-model-cache\") pod \"9afe40ed-514d-4fd3-a64b-4c48549890a3\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " Sep 15 21:09:50.136977 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.136975 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-model-cache\") pod \"3b174a8a-fe9f-4360-b931-0c9016c8832c\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137029 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-home\") pod \"9afe40ed-514d-4fd3-a64b-4c48549890a3\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137068 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xf9j\" (UniqueName: \"kubernetes.io/projected/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-kube-api-access-8xf9j\") pod \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137091 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-home" (OuterVolumeSpecName: "home") pod "3b174a8a-fe9f-4360-b931-0c9016c8832c" (UID: "3b174a8a-fe9f-4360-b931-0c9016c8832c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137106 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-dshm\") pod \"3b174a8a-fe9f-4360-b931-0c9016c8832c\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137153 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-model-cache\") pod \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137202 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-home\") pod \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\" (UID: \"27f9970b-ed43-4a9a-a579-e17ea3b1c5b0\") " Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137231 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-tmp-dir\") pod \"3b174a8a-fe9f-4360-b931-0c9016c8832c\" (UID: \"3b174a8a-fe9f-4360-b931-0c9016c8832c\") " Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137262 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03557738-ca55-4c4d-aa1a-ef659efe0f40-tls-certs\") pod \"03557738-ca55-4c4d-aa1a-ef659efe0f40\" (UID: \"03557738-ca55-4c4d-aa1a-ef659efe0f40\") " Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137299 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jltf2\" (UniqueName: \"kubernetes.io/projected/9afe40ed-514d-4fd3-a64b-4c48549890a3-kube-api-access-jltf2\") pod \"9afe40ed-514d-4fd3-a64b-4c48549890a3\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137468 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-dshm\") pod \"9afe40ed-514d-4fd3-a64b-4c48549890a3\" (UID: \"9afe40ed-514d-4fd3-a64b-4c48549890a3\") " Sep 15 21:09:50.137723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.137678 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" (UID: "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.138280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.138135 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.138280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.138154 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.138280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.138168 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.138280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.138182 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.138280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.138196 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.138549 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.138298 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9afe40ed-514d-4fd3-a64b-4c48549890a3" (UID: "9afe40ed-514d-4fd3-a64b-4c48549890a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.138549 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.138427 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-model-cache" (OuterVolumeSpecName: "model-cache") pod "03557738-ca55-4c4d-aa1a-ef659efe0f40" (UID: "03557738-ca55-4c4d-aa1a-ef659efe0f40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.139137 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.138667 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-home" (OuterVolumeSpecName: "home") pod "9afe40ed-514d-4fd3-a64b-4c48549890a3" (UID: "9afe40ed-514d-4fd3-a64b-4c48549890a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.139137 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.138806 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b174a8a-fe9f-4360-b931-0c9016c8832c" (UID: "3b174a8a-fe9f-4360-b931-0c9016c8832c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.139137 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.139017 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b174a8a-fe9f-4360-b931-0c9016c8832c" (UID: "3b174a8a-fe9f-4360-b931-0c9016c8832c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.139137 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.139019 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" (UID: "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.139137 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.139114 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-home" (OuterVolumeSpecName: "home") pod "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" (UID: "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.139534 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.139474 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afe40ed-514d-4fd3-a64b-4c48549890a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9afe40ed-514d-4fd3-a64b-4c48549890a3" (UID: "9afe40ed-514d-4fd3-a64b-4c48549890a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:09:50.139767 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.139646 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03557738-ca55-4c4d-aa1a-ef659efe0f40-kube-api-access-sq6mz" (OuterVolumeSpecName: "kube-api-access-sq6mz") pod "03557738-ca55-4c4d-aa1a-ef659efe0f40" (UID: "03557738-ca55-4c4d-aa1a-ef659efe0f40"). InnerVolumeSpecName "kube-api-access-sq6mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:50.140609 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.140582 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-dshm" (OuterVolumeSpecName: "dshm") pod "03557738-ca55-4c4d-aa1a-ef659efe0f40" (UID: "03557738-ca55-4c4d-aa1a-ef659efe0f40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.141164 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.141136 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-dshm" (OuterVolumeSpecName: "dshm") pod "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" (UID: "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.141266 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.141138 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b174a8a-fe9f-4360-b931-0c9016c8832c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b174a8a-fe9f-4360-b931-0c9016c8832c" (UID: "3b174a8a-fe9f-4360-b931-0c9016c8832c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:09:50.141556 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.141537 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-dshm" (OuterVolumeSpecName: "dshm") pod "3b174a8a-fe9f-4360-b931-0c9016c8832c" (UID: "3b174a8a-fe9f-4360-b931-0c9016c8832c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.141675 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.141638 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-dshm" (OuterVolumeSpecName: "dshm") pod "9afe40ed-514d-4fd3-a64b-4c48549890a3" (UID: "9afe40ed-514d-4fd3-a64b-4c48549890a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:50.141778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.141683 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" (UID: "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:09:50.141778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.141693 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b174a8a-fe9f-4360-b931-0c9016c8832c-kube-api-access-hd9g7" (OuterVolumeSpecName: "kube-api-access-hd9g7") pod "3b174a8a-fe9f-4360-b931-0c9016c8832c" (UID: "3b174a8a-fe9f-4360-b931-0c9016c8832c"). InnerVolumeSpecName "kube-api-access-hd9g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:50.141778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.141703 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03557738-ca55-4c4d-aa1a-ef659efe0f40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03557738-ca55-4c4d-aa1a-ef659efe0f40" (UID: "03557738-ca55-4c4d-aa1a-ef659efe0f40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:09:50.141778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.141726 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afe40ed-514d-4fd3-a64b-4c48549890a3-kube-api-access-jltf2" (OuterVolumeSpecName: "kube-api-access-jltf2") pod "9afe40ed-514d-4fd3-a64b-4c48549890a3" (UID: "9afe40ed-514d-4fd3-a64b-4c48549890a3"). InnerVolumeSpecName "kube-api-access-jltf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:50.142138 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.142124 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-kube-api-access-8xf9j" (OuterVolumeSpecName: "kube-api-access-8xf9j") pod "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" (UID: "27f9970b-ed43-4a9a-a579-e17ea3b1c5b0"). InnerVolumeSpecName "kube-api-access-8xf9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:50.238925 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238835 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.238925 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238866 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.238925 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238878 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq6mz\" (UniqueName: \"kubernetes.io/projected/03557738-ca55-4c4d-aa1a-ef659efe0f40-kube-api-access-sq6mz\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.238925 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238887 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b174a8a-fe9f-4360-b931-0c9016c8832c-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.238925 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238894 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afe40ed-514d-4fd3-a64b-4c48549890a3-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.238925 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238902 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.238925 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238910 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.238925 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238917 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd9g7\" (UniqueName: \"kubernetes.io/projected/3b174a8a-fe9f-4360-b931-0c9016c8832c-kube-api-access-hd9g7\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.238925 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238926 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03557738-ca55-4c4d-aa1a-ef659efe0f40-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.238925 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238934 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.239355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238941 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.239355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238948 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afe40ed-514d-4fd3-a64b-4c48549890a3-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.239355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238955 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xf9j\" (UniqueName: \"kubernetes.io/projected/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-kube-api-access-8xf9j\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.239355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238963 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.239355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238970 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.239355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238978 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.239355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238985 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b174a8a-fe9f-4360-b931-0c9016c8832c-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.239355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.238992 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03557738-ca55-4c4d-aa1a-ef659efe0f40-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.239355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.239000 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jltf2\" (UniqueName: \"kubernetes.io/projected/9afe40ed-514d-4fd3-a64b-4c48549890a3-kube-api-access-jltf2\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:09:50.381387 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.381362 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr"] Sep 15 21:09:50.384344 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.384321 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-w4-16bde4-kserve-64c4998bcf-l74dr"] Sep 15 21:09:50.396547 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.396518 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz"] Sep 15 21:09:50.400339 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.400316 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/distinct-2-w4-16bde4-kserve-74df8bfc7b-9pthz"] Sep 15 21:09:50.410210 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.410187 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw"] Sep 15 21:09:50.412940 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.412919 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/simulator-w4-16bde4-kserve-7875d6c6fc-5mtcw"] Sep 15 21:09:50.422746 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.422724 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n"] Sep 15 21:09:50.428560 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:50.428537 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/premium-w4-16bde4-kserve-5dc46c8976-sw46n"] Sep 15 21:09:51.311451 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:51.311409 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03557738-ca55-4c4d-aa1a-ef659efe0f40" path="/var/lib/kubelet/pods/03557738-ca55-4c4d-aa1a-ef659efe0f40/volumes" Sep 15 21:09:51.312047 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:51.312026 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" path="/var/lib/kubelet/pods/27f9970b-ed43-4a9a-a579-e17ea3b1c5b0/volumes" Sep 15 21:09:51.312440 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:51.312421 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b174a8a-fe9f-4360-b931-0c9016c8832c" path="/var/lib/kubelet/pods/3b174a8a-fe9f-4360-b931-0c9016c8832c/volumes" Sep 15 21:09:51.313144 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:09:51.313123 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9afe40ed-514d-4fd3-a64b-4c48549890a3" path="/var/lib/kubelet/pods/9afe40ed-514d-4fd3-a64b-4c48549890a3/volumes" Sep 15 21:10:15.876859 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.876824 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq"] Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877169 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" containerName="main" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877180 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" containerName="main" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877191 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22225a81-58b1-4598-81ee-0c49eff2b0ea" containerName="revoke-keys" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877198 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="22225a81-58b1-4598-81ee-0c49eff2b0ea" containerName="revoke-keys" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877209 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b174a8a-fe9f-4360-b931-0c9016c8832c" containerName="main" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877215 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b174a8a-fe9f-4360-b931-0c9016c8832c" containerName="main" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877225 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bfd3d03-d232-4d83-a394-7e75e3db412d" containerName="main" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877230 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bfd3d03-d232-4d83-a394-7e75e3db412d" containerName="main" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877236 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce77d63-7610-47c6-ae43-64b4eb7da4bd" containerName="payload-processing" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877241 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce77d63-7610-47c6-ae43-64b4eb7da4bd" containerName="payload-processing" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877247 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03557738-ca55-4c4d-aa1a-ef659efe0f40" containerName="main" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877252 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="03557738-ca55-4c4d-aa1a-ef659efe0f40" containerName="main" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877263 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9afe40ed-514d-4fd3-a64b-4c48549890a3" containerName="main" Sep 15 21:10:15.877280 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877267 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9afe40ed-514d-4fd3-a64b-4c48549890a3" containerName="main" Sep 15 21:10:15.877746 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877354 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bfd3d03-d232-4d83-a394-7e75e3db412d" containerName="main" Sep 15 21:10:15.877746 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877362 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="22225a81-58b1-4598-81ee-0c49eff2b0ea" containerName="revoke-keys" Sep 15 21:10:15.877746 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877371 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b174a8a-fe9f-4360-b931-0c9016c8832c" containerName="main" Sep 15 21:10:15.877746 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877377 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f9970b-ed43-4a9a-a579-e17ea3b1c5b0" containerName="main" Sep 15 21:10:15.877746 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877383 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9afe40ed-514d-4fd3-a64b-4c48549890a3" containerName="main" Sep 15 21:10:15.877746 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877388 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="03557738-ca55-4c4d-aa1a-ef659efe0f40" containerName="main" Sep 15 21:10:15.877746 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.877393 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce77d63-7610-47c6-ae43-64b4eb7da4bd" containerName="payload-processing" Sep 15 21:10:15.880457 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.880433 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:15.883687 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.883651 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"sub-test-model-728877-kserve-self-signed-certs\"" Sep 15 21:10:15.883826 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.883708 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"kube-root-ca.crt\"" Sep 15 21:10:15.883826 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.883651 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"openshift-service-ca.crt\"" Sep 15 21:10:15.883972 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.883952 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"default-dockercfg-dm9lj\"" Sep 15 21:10:15.889850 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.889822 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq"] Sep 15 21:10:15.969899 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.969848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-home\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:15.970092 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.969955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqxm8\" (UniqueName: \"kubernetes.io/projected/14a1bcd2-120d-4551-9b3f-321d728e3978-kube-api-access-fqxm8\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:15.970092 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.969986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-model-cache\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:15.970092 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.970012 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-tmp-dir\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:15.970092 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.970039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-dshm\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:15.970234 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:15.970148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a1bcd2-120d-4551-9b3f-321d728e3978-tls-certs\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.071308 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.071268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a1bcd2-120d-4551-9b3f-321d728e3978-tls-certs\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.071564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.071342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-home\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.071564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.071415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqxm8\" (UniqueName: \"kubernetes.io/projected/14a1bcd2-120d-4551-9b3f-321d728e3978-kube-api-access-fqxm8\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.071564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.071451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-model-cache\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.071564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.071477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-tmp-dir\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.071564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.071553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-dshm\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.072030 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.072006 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-home\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.072030 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.072020 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-tmp-dir\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.072265 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.072245 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-model-cache\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.074080 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.074053 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a1bcd2-120d-4551-9b3f-321d728e3978-tls-certs\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.074188 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.074051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-dshm\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.081096 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.081069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqxm8\" (UniqueName: \"kubernetes.io/projected/14a1bcd2-120d-4551-9b3f-321d728e3978-kube-api-access-fqxm8\") pod \"sub-test-model-728877-kserve-56b596dc57-x44cq\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.191427 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.191331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:16.466102 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:16.466073 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq"] Sep 15 21:10:16.468301 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:10:16.468271 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14a1bcd2_120d_4551_9b3f_321d728e3978.slice/crio-116167b5fd96ed537b32505ed586ada0c67cee274443f1fc30cf71f0bbdf8720 WatchSource:0}: Error finding container 116167b5fd96ed537b32505ed586ada0c67cee274443f1fc30cf71f0bbdf8720: Status 404 returned error can't find the container with id 116167b5fd96ed537b32505ed586ada0c67cee274443f1fc30cf71f0bbdf8720 Sep 15 21:10:17.211102 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:17.211062 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" event={"ID":"14a1bcd2-120d-4551-9b3f-321d728e3978","Type":"ContainerStarted","Data":"09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a"} Sep 15 21:10:17.211553 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:17.211115 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" event={"ID":"14a1bcd2-120d-4551-9b3f-321d728e3978","Type":"ContainerStarted","Data":"116167b5fd96ed537b32505ed586ada0c67cee274443f1fc30cf71f0bbdf8720"} Sep 15 21:10:17.227866 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:17.227795 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" podStartSLOduration=2.227772828 podStartE2EDuration="2.227772828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:17.22732828 +0000 UTC m=+2014.421281737" watchObservedRunningTime="2026-09-15 21:10:17.227772828 +0000 UTC m=+2014.421726285" Sep 15 21:10:26.191906 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:26.191867 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:26.192311 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:26.192114 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:26.203972 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:26.203943 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:26.272313 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:26.272281 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:10:43.781931 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:43.781898 2579 scope.go:117] "RemoveContainer" containerID="f5f45b168bbe5758d0e06f520933f72bc24c1e5878bed9b9ba7666a13b674861" Sep 15 21:10:51.175767 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:10:51.175725 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq"] Sep 15 21:11:06.198028 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.197910 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" podUID="14a1bcd2-120d-4551-9b3f-321d728e3978" containerName="main" containerID="cri-o://09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a" gracePeriod=45 Sep 15 21:11:06.262278 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.262232 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" podUID="14a1bcd2-120d-4551-9b3f-321d728e3978" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": dial tcp 10.132.0.96:8000: connect: connection refused" Sep 15 21:11:06.450593 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.450530 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:11:06.465893 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.465861 2579 generic.go:356] "Generic (PLEG): container finished" podID="14a1bcd2-120d-4551-9b3f-321d728e3978" containerID="09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a" exitCode=0 Sep 15 21:11:06.466034 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.465907 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" Sep 15 21:11:06.466034 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.465951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" event={"ID":"14a1bcd2-120d-4551-9b3f-321d728e3978","Type":"ContainerDied","Data":"09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a"} Sep 15 21:11:06.466034 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.465996 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq" event={"ID":"14a1bcd2-120d-4551-9b3f-321d728e3978","Type":"ContainerDied","Data":"116167b5fd96ed537b32505ed586ada0c67cee274443f1fc30cf71f0bbdf8720"} Sep 15 21:11:06.466034 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.466016 2579 scope.go:117] "RemoveContainer" containerID="09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a" Sep 15 21:11:06.475737 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.475718 2579 scope.go:117] "RemoveContainer" containerID="09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a" Sep 15 21:11:06.476020 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:11:06.475997 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a\": container with ID starting with 09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a not found: ID does not exist" containerID="09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a" Sep 15 21:11:06.476115 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.476027 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a"} err="failed to get container status \"09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a\": rpc error: code = NotFound desc = could not find container \"09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a\": container with ID starting with 09ca7ed612453664c764fc7f7d0bd2275af422bad6ce3b0e99fcd2f71dbbdc6a not found: ID does not exist" Sep 15 21:11:06.547798 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.547760 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a1bcd2-120d-4551-9b3f-321d728e3978-tls-certs\") pod \"14a1bcd2-120d-4551-9b3f-321d728e3978\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " Sep 15 21:11:06.547798 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.547802 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-home\") pod \"14a1bcd2-120d-4551-9b3f-321d728e3978\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " Sep 15 21:11:06.548017 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.547836 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-dshm\") pod \"14a1bcd2-120d-4551-9b3f-321d728e3978\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " Sep 15 21:11:06.548017 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.547944 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-model-cache\") pod \"14a1bcd2-120d-4551-9b3f-321d728e3978\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " Sep 15 21:11:06.548111 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.548048 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqxm8\" (UniqueName: \"kubernetes.io/projected/14a1bcd2-120d-4551-9b3f-321d728e3978-kube-api-access-fqxm8\") pod \"14a1bcd2-120d-4551-9b3f-321d728e3978\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " Sep 15 21:11:06.548111 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.548081 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-home" (OuterVolumeSpecName: "home") pod "14a1bcd2-120d-4551-9b3f-321d728e3978" (UID: "14a1bcd2-120d-4551-9b3f-321d728e3978"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:11:06.548201 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.548108 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-tmp-dir\") pod \"14a1bcd2-120d-4551-9b3f-321d728e3978\" (UID: \"14a1bcd2-120d-4551-9b3f-321d728e3978\") " Sep 15 21:11:06.548201 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.548167 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-model-cache" (OuterVolumeSpecName: "model-cache") pod "14a1bcd2-120d-4551-9b3f-321d728e3978" (UID: "14a1bcd2-120d-4551-9b3f-321d728e3978"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:11:06.548347 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.548323 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14a1bcd2-120d-4551-9b3f-321d728e3978" (UID: "14a1bcd2-120d-4551-9b3f-321d728e3978"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:11:06.548527 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.548476 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:11:06.548654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.548592 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:11:06.548697 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.548675 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:11:06.550246 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.550219 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-dshm" (OuterVolumeSpecName: "dshm") pod "14a1bcd2-120d-4551-9b3f-321d728e3978" (UID: "14a1bcd2-120d-4551-9b3f-321d728e3978"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:11:06.550349 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.550260 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a1bcd2-120d-4551-9b3f-321d728e3978-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14a1bcd2-120d-4551-9b3f-321d728e3978" (UID: "14a1bcd2-120d-4551-9b3f-321d728e3978"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:11:06.550394 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.550344 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a1bcd2-120d-4551-9b3f-321d728e3978-kube-api-access-fqxm8" (OuterVolumeSpecName: "kube-api-access-fqxm8") pod "14a1bcd2-120d-4551-9b3f-321d728e3978" (UID: "14a1bcd2-120d-4551-9b3f-321d728e3978"). InnerVolumeSpecName "kube-api-access-fqxm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:11:06.649735 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.649697 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a1bcd2-120d-4551-9b3f-321d728e3978-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:11:06.649735 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.649729 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a1bcd2-120d-4551-9b3f-321d728e3978-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:11:06.649735 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.649740 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqxm8\" (UniqueName: \"kubernetes.io/projected/14a1bcd2-120d-4551-9b3f-321d728e3978-kube-api-access-fqxm8\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:11:06.785834 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.785800 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq"] Sep 15 21:11:06.789775 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:06.789749 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/sub-test-model-728877-kserve-56b596dc57-x44cq"] Sep 15 21:11:07.312765 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:07.312736 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a1bcd2-120d-4551-9b3f-321d728e3978" path="/var/lib/kubelet/pods/14a1bcd2-120d-4551-9b3f-321d728e3978/volumes" Sep 15 21:11:11.169577 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.169477 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4"] Sep 15 21:11:11.170036 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.169866 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a1bcd2-120d-4551-9b3f-321d728e3978" containerName="main" Sep 15 21:11:11.170036 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.169877 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a1bcd2-120d-4551-9b3f-321d728e3978" containerName="main" Sep 15 21:11:11.170036 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.169955 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a1bcd2-120d-4551-9b3f-321d728e3978" containerName="main" Sep 15 21:11:11.174384 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.174360 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.177208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.177180 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"default-dockercfg-m9ttf\"" Sep 15 21:11:11.177208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.177199 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"openshift-service-ca.crt\"" Sep 15 21:11:11.177459 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.177244 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"kube-root-ca.crt\"" Sep 15 21:11:11.177459 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.177192 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"rate-test-model-280fb4-kserve-self-signed-certs\"" Sep 15 21:11:11.180277 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.180256 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4"] Sep 15 21:11:11.292325 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.292288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-home\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.292552 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.292359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-model-cache\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.292552 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.292380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tmp-dir\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.292552 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.292439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-dshm\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.292552 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.292516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tls-certs\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.292698 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.292561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnflk\" (UniqueName: \"kubernetes.io/projected/fc78ee47-7ce1-4539-93bc-ca75af6b151a-kube-api-access-fnflk\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.394004 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.393953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tls-certs\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.394004 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.394005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnflk\" (UniqueName: \"kubernetes.io/projected/fc78ee47-7ce1-4539-93bc-ca75af6b151a-kube-api-access-fnflk\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.394268 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.394051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-home\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.394268 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.394098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-model-cache\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.394268 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.394114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tmp-dir\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.394268 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.394245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-dshm\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.394600 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.394579 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-model-cache\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.394600 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.394592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tmp-dir\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.394711 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.394649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-home\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.396579 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.396551 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-dshm\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.396724 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.396704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tls-certs\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.402234 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.402211 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnflk\" (UniqueName: \"kubernetes.io/projected/fc78ee47-7ce1-4539-93bc-ca75af6b151a-kube-api-access-fnflk\") pod \"rate-test-model-280fb4-kserve-9d595d779-29gn4\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.485753 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.485668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:11.768393 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:11.768370 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4"] Sep 15 21:11:11.770678 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:11:11.770636 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc78ee47_7ce1_4539_93bc_ca75af6b151a.slice/crio-1491b077f3baa04ecfddfe714d982008b2e6abc0ea48890fe660bcab5fd2028b WatchSource:0}: Error finding container 1491b077f3baa04ecfddfe714d982008b2e6abc0ea48890fe660bcab5fd2028b: Status 404 returned error can't find the container with id 1491b077f3baa04ecfddfe714d982008b2e6abc0ea48890fe660bcab5fd2028b Sep 15 21:11:12.503594 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:12.503559 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" event={"ID":"fc78ee47-7ce1-4539-93bc-ca75af6b151a","Type":"ContainerStarted","Data":"b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc"} Sep 15 21:11:12.504078 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:12.503609 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" event={"ID":"fc78ee47-7ce1-4539-93bc-ca75af6b151a","Type":"ContainerStarted","Data":"1491b077f3baa04ecfddfe714d982008b2e6abc0ea48890fe660bcab5fd2028b"} Sep 15 21:11:12.519559 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:12.519482 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" podStartSLOduration=1.519464942 podStartE2EDuration="1.519464942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:12.519059823 +0000 UTC m=+2069.713013306" watchObservedRunningTime="2026-09-15 21:11:12.519464942 +0000 UTC m=+2069.713418466" Sep 15 21:11:21.486063 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:21.486028 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:21.486697 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:21.486248 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:21.498892 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:21.498866 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:21.560809 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:21.560772 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:11:26.368476 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.368439 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x"] Sep 15 21:11:26.371945 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.371917 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.375113 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.375092 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"default-dockercfg-dm9lj\"" Sep 15 21:11:26.375257 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.375111 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"kube-root-ca.crt\"" Sep 15 21:11:26.375257 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.375111 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"rate-test-model-728877-kserve-self-signed-certs\"" Sep 15 21:11:26.375257 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.375091 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"openshift-service-ca.crt\"" Sep 15 21:11:26.383223 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.383200 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x"] Sep 15 21:11:26.541524 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.541463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-tmp-dir\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.541709 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.541564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cc65aa-7428-4205-b7d1-3f78a8709faa-tls-certs\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.541709 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.541599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmwd\" (UniqueName: \"kubernetes.io/projected/70cc65aa-7428-4205-b7d1-3f78a8709faa-kube-api-access-vjmwd\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.541709 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.541630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-dshm\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.541709 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.541655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-model-cache\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.541904 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.541725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-home\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.642944 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.642854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-home\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.642944 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.642891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-tmp-dir\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.642944 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.642923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cc65aa-7428-4205-b7d1-3f78a8709faa-tls-certs\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.643202 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.642954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmwd\" (UniqueName: \"kubernetes.io/projected/70cc65aa-7428-4205-b7d1-3f78a8709faa-kube-api-access-vjmwd\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.643202 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.642980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-dshm\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.643202 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.642998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-model-cache\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.643357 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.643331 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-home\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.643405 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.643345 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-tmp-dir\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.643405 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.643390 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-model-cache\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.645473 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.645443 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-dshm\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.645658 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.645642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cc65aa-7428-4205-b7d1-3f78a8709faa-tls-certs\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.649957 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.649933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmwd\" (UniqueName: \"kubernetes.io/projected/70cc65aa-7428-4205-b7d1-3f78a8709faa-kube-api-access-vjmwd\") pod \"rate-test-model-728877-kserve-85fbc9757f-b7c5x\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.683184 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.683149 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:26.954944 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:26.954920 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x"] Sep 15 21:11:26.956921 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:11:26.956889 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70cc65aa_7428_4205_b7d1_3f78a8709faa.slice/crio-d22a4b0c7fa2424f540592a8ca69308619b523ed672b885758744750f978e530 WatchSource:0}: Error finding container d22a4b0c7fa2424f540592a8ca69308619b523ed672b885758744750f978e530: Status 404 returned error can't find the container with id d22a4b0c7fa2424f540592a8ca69308619b523ed672b885758744750f978e530 Sep 15 21:11:27.581167 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:27.581132 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" event={"ID":"70cc65aa-7428-4205-b7d1-3f78a8709faa","Type":"ContainerStarted","Data":"31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5"} Sep 15 21:11:27.581167 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:27.581181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" event={"ID":"70cc65aa-7428-4205-b7d1-3f78a8709faa","Type":"ContainerStarted","Data":"d22a4b0c7fa2424f540592a8ca69308619b523ed672b885758744750f978e530"} Sep 15 21:11:27.599154 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:27.599097 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" podStartSLOduration=1.5990792740000002 podStartE2EDuration="1.599079274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:27.596783473 +0000 UTC m=+2084.790736930" watchObservedRunningTime="2026-09-15 21:11:27.599079274 +0000 UTC m=+2084.793032728" Sep 15 21:11:36.683474 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:36.683437 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:36.683903 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:36.683566 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:36.696180 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:36.696153 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:11:37.644695 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:11:37.644663 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:12:03.896948 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:03.896906 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4"] Sep 15 21:12:04.883392 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:04.883357 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x"] Sep 15 21:12:18.916901 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:18.916842 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" podUID="fc78ee47-7ce1-4539-93bc-ca75af6b151a" containerName="main" containerID="cri-o://b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc" gracePeriod=45 Sep 15 21:12:19.163720 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.163696 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:12:19.229016 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.228936 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnflk\" (UniqueName: \"kubernetes.io/projected/fc78ee47-7ce1-4539-93bc-ca75af6b151a-kube-api-access-fnflk\") pod \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " Sep 15 21:12:19.229016 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.228986 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tmp-dir\") pod \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " Sep 15 21:12:19.229258 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.229033 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-home\") pod \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " Sep 15 21:12:19.229258 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.229100 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tls-certs\") pod \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " Sep 15 21:12:19.229258 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.229135 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-model-cache\") pod \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " Sep 15 21:12:19.229258 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.229172 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-dshm\") pod \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\" (UID: \"fc78ee47-7ce1-4539-93bc-ca75af6b151a\") " Sep 15 21:12:19.229484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.229335 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc78ee47-7ce1-4539-93bc-ca75af6b151a" (UID: "fc78ee47-7ce1-4539-93bc-ca75af6b151a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:12:19.229484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.229387 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-home" (OuterVolumeSpecName: "home") pod "fc78ee47-7ce1-4539-93bc-ca75af6b151a" (UID: "fc78ee47-7ce1-4539-93bc-ca75af6b151a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:12:19.229484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.229406 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc78ee47-7ce1-4539-93bc-ca75af6b151a" (UID: "fc78ee47-7ce1-4539-93bc-ca75af6b151a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:12:19.229626 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.229570 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:19.229626 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.229588 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:19.229626 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.229600 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:19.231315 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.231293 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-dshm" (OuterVolumeSpecName: "dshm") pod "fc78ee47-7ce1-4539-93bc-ca75af6b151a" (UID: "fc78ee47-7ce1-4539-93bc-ca75af6b151a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:12:19.231400 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.231323 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc78ee47-7ce1-4539-93bc-ca75af6b151a" (UID: "fc78ee47-7ce1-4539-93bc-ca75af6b151a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:12:19.231446 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.231402 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc78ee47-7ce1-4539-93bc-ca75af6b151a-kube-api-access-fnflk" (OuterVolumeSpecName: "kube-api-access-fnflk") pod "fc78ee47-7ce1-4539-93bc-ca75af6b151a" (UID: "fc78ee47-7ce1-4539-93bc-ca75af6b151a"). InnerVolumeSpecName "kube-api-access-fnflk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:12:19.330530 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.330486 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc78ee47-7ce1-4539-93bc-ca75af6b151a-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:19.330530 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.330524 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc78ee47-7ce1-4539-93bc-ca75af6b151a-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:19.330530 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.330533 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnflk\" (UniqueName: \"kubernetes.io/projected/fc78ee47-7ce1-4539-93bc-ca75af6b151a-kube-api-access-fnflk\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:19.856476 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.856443 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc78ee47-7ce1-4539-93bc-ca75af6b151a" containerID="b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc" exitCode=0 Sep 15 21:12:19.856697 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.856483 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" Sep 15 21:12:19.856697 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.856529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" event={"ID":"fc78ee47-7ce1-4539-93bc-ca75af6b151a","Type":"ContainerDied","Data":"b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc"} Sep 15 21:12:19.856697 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.856574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4" event={"ID":"fc78ee47-7ce1-4539-93bc-ca75af6b151a","Type":"ContainerDied","Data":"1491b077f3baa04ecfddfe714d982008b2e6abc0ea48890fe660bcab5fd2028b"} Sep 15 21:12:19.856697 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.856592 2579 scope.go:117] "RemoveContainer" containerID="b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc" Sep 15 21:12:19.865564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.865553 2579 scope.go:117] "RemoveContainer" containerID="b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc" Sep 15 21:12:19.865846 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:12:19.865830 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc\": container with ID starting with b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc not found: ID does not exist" containerID="b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc" Sep 15 21:12:19.865896 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.865855 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc"} err="failed to get container status \"b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc\": rpc error: code = NotFound desc = could not find container \"b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc\": container with ID starting with b4fec9c23b7b53d2e5bce2ff321fcd02e4208e215a23be905faba46fbe7cd0bc not found: ID does not exist" Sep 15 21:12:19.872165 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.872141 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4"] Sep 15 21:12:19.875611 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.875591 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/rate-test-model-280fb4-kserve-9d595d779-29gn4"] Sep 15 21:12:19.903090 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:19.903050 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" podUID="70cc65aa-7428-4205-b7d1-3f78a8709faa" containerName="main" containerID="cri-o://31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5" gracePeriod=45 Sep 15 21:12:20.153675 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.153651 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:12:20.237974 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.237939 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-dshm\") pod \"70cc65aa-7428-4205-b7d1-3f78a8709faa\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " Sep 15 21:12:20.238175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.237986 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-model-cache\") pod \"70cc65aa-7428-4205-b7d1-3f78a8709faa\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " Sep 15 21:12:20.238175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.238015 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-tmp-dir\") pod \"70cc65aa-7428-4205-b7d1-3f78a8709faa\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " Sep 15 21:12:20.238175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.238040 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cc65aa-7428-4205-b7d1-3f78a8709faa-tls-certs\") pod \"70cc65aa-7428-4205-b7d1-3f78a8709faa\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " Sep 15 21:12:20.238175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.238144 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-home\") pod \"70cc65aa-7428-4205-b7d1-3f78a8709faa\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " Sep 15 21:12:20.238385 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.238194 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjmwd\" (UniqueName: \"kubernetes.io/projected/70cc65aa-7428-4205-b7d1-3f78a8709faa-kube-api-access-vjmwd\") pod \"70cc65aa-7428-4205-b7d1-3f78a8709faa\" (UID: \"70cc65aa-7428-4205-b7d1-3f78a8709faa\") " Sep 15 21:12:20.238385 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.238254 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-model-cache" (OuterVolumeSpecName: "model-cache") pod "70cc65aa-7428-4205-b7d1-3f78a8709faa" (UID: "70cc65aa-7428-4205-b7d1-3f78a8709faa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:12:20.238385 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.238271 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70cc65aa-7428-4205-b7d1-3f78a8709faa" (UID: "70cc65aa-7428-4205-b7d1-3f78a8709faa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:12:20.238579 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.238427 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-home" (OuterVolumeSpecName: "home") pod "70cc65aa-7428-4205-b7d1-3f78a8709faa" (UID: "70cc65aa-7428-4205-b7d1-3f78a8709faa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:12:20.238579 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.238560 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:20.238694 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.238580 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:20.238694 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.238590 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:20.240199 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.240178 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-dshm" (OuterVolumeSpecName: "dshm") pod "70cc65aa-7428-4205-b7d1-3f78a8709faa" (UID: "70cc65aa-7428-4205-b7d1-3f78a8709faa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:12:20.240381 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.240366 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70cc65aa-7428-4205-b7d1-3f78a8709faa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70cc65aa-7428-4205-b7d1-3f78a8709faa" (UID: "70cc65aa-7428-4205-b7d1-3f78a8709faa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:12:20.240460 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.240441 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cc65aa-7428-4205-b7d1-3f78a8709faa-kube-api-access-vjmwd" (OuterVolumeSpecName: "kube-api-access-vjmwd") pod "70cc65aa-7428-4205-b7d1-3f78a8709faa" (UID: "70cc65aa-7428-4205-b7d1-3f78a8709faa"). InnerVolumeSpecName "kube-api-access-vjmwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:12:20.339692 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.339660 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjmwd\" (UniqueName: \"kubernetes.io/projected/70cc65aa-7428-4205-b7d1-3f78a8709faa-kube-api-access-vjmwd\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:20.339692 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.339688 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70cc65aa-7428-4205-b7d1-3f78a8709faa-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:20.339692 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.339698 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cc65aa-7428-4205-b7d1-3f78a8709faa-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:12:20.864542 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.864488 2579 generic.go:356] "Generic (PLEG): container finished" podID="70cc65aa-7428-4205-b7d1-3f78a8709faa" containerID="31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5" exitCode=0 Sep 15 21:12:20.864713 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.864555 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" Sep 15 21:12:20.864713 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.864569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" event={"ID":"70cc65aa-7428-4205-b7d1-3f78a8709faa","Type":"ContainerDied","Data":"31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5"} Sep 15 21:12:20.864713 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.864615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x" event={"ID":"70cc65aa-7428-4205-b7d1-3f78a8709faa","Type":"ContainerDied","Data":"d22a4b0c7fa2424f540592a8ca69308619b523ed672b885758744750f978e530"} Sep 15 21:12:20.864713 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.864634 2579 scope.go:117] "RemoveContainer" containerID="31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5" Sep 15 21:12:20.874050 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.874029 2579 scope.go:117] "RemoveContainer" containerID="31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5" Sep 15 21:12:20.874310 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:12:20.874293 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5\": container with ID starting with 31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5 not found: ID does not exist" containerID="31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5" Sep 15 21:12:20.874351 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.874320 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5"} err="failed to get container status \"31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5\": rpc error: code = NotFound desc = could not find container \"31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5\": container with ID starting with 31feac01aca244c09b56c9fe73e9779716dcfc55ab569f30a9c037c75bacc5e5 not found: ID does not exist" Sep 15 21:12:20.885372 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.885350 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x"] Sep 15 21:12:20.888947 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:20.888922 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/rate-test-model-728877-kserve-85fbc9757f-b7c5x"] Sep 15 21:12:21.311096 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:21.311065 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70cc65aa-7428-4205-b7d1-3f78a8709faa" path="/var/lib/kubelet/pods/70cc65aa-7428-4205-b7d1-3f78a8709faa/volumes" Sep 15 21:12:21.311508 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:21.311480 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc78ee47-7ce1-4539-93bc-ca75af6b151a" path="/var/lib/kubelet/pods/fc78ee47-7ce1-4539-93bc-ca75af6b151a/volumes" Sep 15 21:12:34.631028 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.630947 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8"] Sep 15 21:12:34.631530 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.631456 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc78ee47-7ce1-4539-93bc-ca75af6b151a" containerName="main" Sep 15 21:12:34.631530 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.631475 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc78ee47-7ce1-4539-93bc-ca75af6b151a" containerName="main" Sep 15 21:12:34.631530 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.631526 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70cc65aa-7428-4205-b7d1-3f78a8709faa" containerName="main" Sep 15 21:12:34.631715 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.631535 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70cc65aa-7428-4205-b7d1-3f78a8709faa" containerName="main" Sep 15 21:12:34.631715 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.631644 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc78ee47-7ce1-4539-93bc-ca75af6b151a" containerName="main" Sep 15 21:12:34.631715 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.631657 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70cc65aa-7428-4205-b7d1-3f78a8709faa" containerName="main" Sep 15 21:12:34.636048 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.636026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.638296 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.638275 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7f09a7-openshift-default-dockercfg-t565k\"" Sep 15 21:12:34.651551 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.645829 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8"] Sep 15 21:12:34.768449 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.768406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-socket\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.768449 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.768444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-credential-socket\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.768710 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.768466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-envoy\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.768710 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.768586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700944ff-55e6-462f-95b0-1222e07aa47f-istio-podinfo\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.768710 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.768641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fggbc\" (UniqueName: \"kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-kube-api-access-fggbc\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.768710 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.768708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-istio-token\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.768862 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.768737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-data\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.768862 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.768757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700944ff-55e6-462f-95b0-1222e07aa47f-istiod-ca-cert\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.768862 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.768780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-certs\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870021 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.869983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-istio-token\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870021 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-data\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870298 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700944ff-55e6-462f-95b0-1222e07aa47f-istiod-ca-cert\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870298 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-certs\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870298 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-socket\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870298 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-credential-socket\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870298 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-envoy\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870604 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700944ff-55e6-462f-95b0-1222e07aa47f-istio-podinfo\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870604 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fggbc\" (UniqueName: \"kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-kube-api-access-fggbc\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870604 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870399 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-data\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870604 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870542 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-certs\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870801 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-socket\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870801 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-credential-socket\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.870801 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.870789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700944ff-55e6-462f-95b0-1222e07aa47f-istiod-ca-cert\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.872931 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.872901 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-envoy\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.873061 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.873009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700944ff-55e6-462f-95b0-1222e07aa47f-istio-podinfo\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.881491 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.881425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-istio-token\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.881649 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.881622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fggbc\" (UniqueName: \"kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-kube-api-access-fggbc\") pod \"e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:34.948511 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:34.948475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:35.225340 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:35.225304 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8"] Sep 15 21:12:35.228006 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:12:35.227978 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700944ff_55e6_462f_95b0_1222e07aa47f.slice/crio-5d87e51dd052317210253ee5df6d4063f4312c0e0b8da8e35b35e1fcd44a758b WatchSource:0}: Error finding container 5d87e51dd052317210253ee5df6d4063f4312c0e0b8da8e35b35e1fcd44a758b: Status 404 returned error can't find the container with id 5d87e51dd052317210253ee5df6d4063f4312c0e0b8da8e35b35e1fcd44a758b Sep 15 21:12:35.230746 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:35.230708 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:12:35.230816 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:35.230799 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:12:35.230858 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:35.230841 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:12:35.947008 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:35.946967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" event={"ID":"700944ff-55e6-462f-95b0-1222e07aa47f","Type":"ContainerStarted","Data":"29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e"} Sep 15 21:12:35.947422 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:35.947023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" event={"ID":"700944ff-55e6-462f-95b0-1222e07aa47f","Type":"ContainerStarted","Data":"5d87e51dd052317210253ee5df6d4063f4312c0e0b8da8e35b35e1fcd44a758b"} Sep 15 21:12:35.949243 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:35.949223 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:35.968374 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:35.968331 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" podStartSLOduration=1.968316875 podStartE2EDuration="1.968316875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:35.96574503 +0000 UTC m=+2153.159698485" watchObservedRunningTime="2026-09-15 21:12:35.968316875 +0000 UTC m=+2153.162270332" Sep 15 21:12:36.954110 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:36.954081 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:37.957614 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:37.957585 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:37.958547 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:37.958477 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:41.220637 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.220603 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw"] Sep 15 21:12:41.224182 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.224165 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:12:41.226361 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.226338 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7f09a7-dockercfg-k28nv\"" Sep 15 21:12:41.226468 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.226405 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7f09a7\"" Sep 15 21:12:41.236196 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.236147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw"] Sep 15 21:12:41.250025 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.249998 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2"] Sep 15 21:12:41.253426 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.253407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:12:41.265446 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.265423 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2"] Sep 15 21:12:41.335076 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.335044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93108ba3-b5da-4f39-bdc1-70c2ff12f454-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw\" (UID: \"93108ba3-b5da-4f39-bdc1-70c2ff12f454\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:12:41.335263 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.335105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsvm2\" (UniqueName: \"kubernetes.io/projected/93108ba3-b5da-4f39-bdc1-70c2ff12f454-kube-api-access-nsvm2\") pod \"payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw\" (UID: \"93108ba3-b5da-4f39-bdc1-70c2ff12f454\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:12:41.335263 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.335133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec362713-c4d6-4cf1-b223-62c15e950f84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2\" (UID: \"ec362713-c4d6-4cf1-b223-62c15e950f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:12:41.335263 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.335217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcbpz\" (UniqueName: \"kubernetes.io/projected/ec362713-c4d6-4cf1-b223-62c15e950f84-kube-api-access-zcbpz\") pod \"payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2\" (UID: \"ec362713-c4d6-4cf1-b223-62c15e950f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:12:41.436382 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.436344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93108ba3-b5da-4f39-bdc1-70c2ff12f454-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw\" (UID: \"93108ba3-b5da-4f39-bdc1-70c2ff12f454\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:12:41.436603 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.436394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsvm2\" (UniqueName: \"kubernetes.io/projected/93108ba3-b5da-4f39-bdc1-70c2ff12f454-kube-api-access-nsvm2\") pod \"payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw\" (UID: \"93108ba3-b5da-4f39-bdc1-70c2ff12f454\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:12:41.436603 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.436415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec362713-c4d6-4cf1-b223-62c15e950f84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2\" (UID: \"ec362713-c4d6-4cf1-b223-62c15e950f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:12:41.436603 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.436469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcbpz\" (UniqueName: \"kubernetes.io/projected/ec362713-c4d6-4cf1-b223-62c15e950f84-kube-api-access-zcbpz\") pod \"payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2\" (UID: \"ec362713-c4d6-4cf1-b223-62c15e950f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:12:41.437155 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.437134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec362713-c4d6-4cf1-b223-62c15e950f84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2\" (UID: \"ec362713-c4d6-4cf1-b223-62c15e950f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:12:41.437222 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.437203 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93108ba3-b5da-4f39-bdc1-70c2ff12f454-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw\" (UID: \"93108ba3-b5da-4f39-bdc1-70c2ff12f454\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:12:41.445250 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.445218 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsvm2\" (UniqueName: \"kubernetes.io/projected/93108ba3-b5da-4f39-bdc1-70c2ff12f454-kube-api-access-nsvm2\") pod \"payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw\" (UID: \"93108ba3-b5da-4f39-bdc1-70c2ff12f454\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:12:41.445421 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.445399 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcbpz\" (UniqueName: \"kubernetes.io/projected/ec362713-c4d6-4cf1-b223-62c15e950f84-kube-api-access-zcbpz\") pod \"payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2\" (UID: \"ec362713-c4d6-4cf1-b223-62c15e950f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:12:41.535163 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.535130 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:12:41.564040 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.564011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:12:41.809251 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.809217 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw"] Sep 15 21:12:41.811595 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:12:41.811570 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93108ba3_b5da_4f39_bdc1_70c2ff12f454.slice/crio-1796b29b0de83603a8b90375023f8c0eaf7d52430d69ca8b88bb830fd38894b4 WatchSource:0}: Error finding container 1796b29b0de83603a8b90375023f8c0eaf7d52430d69ca8b88bb830fd38894b4: Status 404 returned error can't find the container with id 1796b29b0de83603a8b90375023f8c0eaf7d52430d69ca8b88bb830fd38894b4 Sep 15 21:12:41.848760 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.848737 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2"] Sep 15 21:12:41.850847 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:12:41.850820 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec362713_c4d6_4cf1_b223_62c15e950f84.slice/crio-19fbb46515e979a204fa75397ee617527cd89b62dad5e5b514be04d12cb98dc1 WatchSource:0}: Error finding container 19fbb46515e979a204fa75397ee617527cd89b62dad5e5b514be04d12cb98dc1: Status 404 returned error can't find the container with id 19fbb46515e979a204fa75397ee617527cd89b62dad5e5b514be04d12cb98dc1 Sep 15 21:12:41.980572 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.980401 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" event={"ID":"93108ba3-b5da-4f39-bdc1-70c2ff12f454","Type":"ContainerStarted","Data":"8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421"} Sep 15 21:12:41.980572 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.980472 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" event={"ID":"93108ba3-b5da-4f39-bdc1-70c2ff12f454","Type":"ContainerStarted","Data":"1796b29b0de83603a8b90375023f8c0eaf7d52430d69ca8b88bb830fd38894b4"} Sep 15 21:12:41.980930 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.980597 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:12:41.982308 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.982280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" event={"ID":"ec362713-c4d6-4cf1-b223-62c15e950f84","Type":"ContainerStarted","Data":"c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55"} Sep 15 21:12:41.982440 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.982329 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" event={"ID":"ec362713-c4d6-4cf1-b223-62c15e950f84","Type":"ContainerStarted","Data":"19fbb46515e979a204fa75397ee617527cd89b62dad5e5b514be04d12cb98dc1"} Sep 15 21:12:41.982440 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.982357 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:12:41.996805 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:41.996758 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" podStartSLOduration=0.99674302 podStartE2EDuration="996.74302ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:41.995145294 +0000 UTC m=+2159.189098749" watchObservedRunningTime="2026-09-15 21:12:41.99674302 +0000 UTC m=+2159.190696486" Sep 15 21:12:42.012264 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:42.012217 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" podStartSLOduration=1.012202798 podStartE2EDuration="1.012202798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:42.010896438 +0000 UTC m=+2159.204849892" watchObservedRunningTime="2026-09-15 21:12:42.012202798 +0000 UTC m=+2159.206156252" Sep 15 21:12:52.991376 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:52.991343 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:12:52.991846 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:52.991430 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:12:53.999755 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:53.999714 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 15 21:12:54.000147 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:53.999795 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 15 21:12:56.060189 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:56.060154 2579 generic.go:356] "Generic (PLEG): container finished" podID="700944ff-55e6-462f-95b0-1222e07aa47f" containerID="29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e" exitCode=0 Sep 15 21:12:56.060641 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:56.060229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" event={"ID":"700944ff-55e6-462f-95b0-1222e07aa47f","Type":"ContainerDied","Data":"29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e"} Sep 15 21:12:56.060691 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:56.060675 2579 scope.go:117] "RemoveContainer" containerID="29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e" Sep 15 21:12:56.061890 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:56.061861 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:12:56.061965 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:56.061931 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:12:56.062005 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:56.061968 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:12:57.068096 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:57.068059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" event={"ID":"700944ff-55e6-462f-95b0-1222e07aa47f","Type":"ContainerStarted","Data":"39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af"} Sep 15 21:12:57.949289 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:57.949252 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:12:57.949508 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:57.949473 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Sep 15 21:12:57.949564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:57.949541 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Sep 15 21:12:58.949476 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:58.949440 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Sep 15 21:12:58.949929 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:58.949529 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Sep 15 21:12:59.949178 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:59.949133 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Sep 15 21:12:59.949355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:12:59.949219 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Sep 15 21:13:00.948931 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:00.948891 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Sep 15 21:13:00.949372 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:00.948953 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Sep 15 21:13:01.949200 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:01.949161 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Sep 15 21:13:01.949630 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:01.949254 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Sep 15 21:13:02.952769 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:02.952738 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:13:02.953217 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:02.953031 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:13:02.953871 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:02.953850 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:13:13.183941 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:13.183909 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2_ec362713-c4d6-4cf1-b223-62c15e950f84/payload-processing/0.log" Sep 15 21:13:43.867868 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:43.867837 2579 scope.go:117] "RemoveContainer" containerID="6ea00636c0df9cc93f35a9b2d9ca025feb0cd738842f119b874ad841abcc73d5" Sep 15 21:13:44.079029 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.079000 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2_ec362713-c4d6-4cf1-b223-62c15e950f84/payload-processing/0.log" Sep 15 21:13:44.620711 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.620672 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd"] Sep 15 21:13:44.625696 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.625673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.628007 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.627984 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-a6f49f-openshift-default-dockercfg-mfpqk\"" Sep 15 21:13:44.635857 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.635832 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd"] Sep 15 21:13:44.663871 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.663839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-socket\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.664011 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.663892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-envoy\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.664011 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.663913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.664011 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.663977 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-certs\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.664123 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.664021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnz9z\" (UniqueName: \"kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-kube-api-access-pnz9z\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.664123 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.664045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-credential-socket\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.664123 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.664102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-data\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.664225 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.664128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-token\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.664225 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.664153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-podinfo\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.765362 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-envoy\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.765362 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.765638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-certs\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.765638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnz9z\" (UniqueName: \"kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-kube-api-access-pnz9z\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.765638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-credential-socket\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.765638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-data\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.765638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-token\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.765638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-podinfo\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.765638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-socket\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.766015 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-credential-socket\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.766015 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.765962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-socket\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.766091 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.766031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-data\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.766161 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.766141 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-certs\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.766209 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.766188 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.767873 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.767852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-envoy\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.768116 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.768099 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-podinfo\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.777340 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.777314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-token\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.777447 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.777367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnz9z\" (UniqueName: \"kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-kube-api-access-pnz9z\") pod \"e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:44.937524 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:44.937422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:45.216680 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:45.216652 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd"] Sep 15 21:13:45.218820 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:13:45.218795 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc101f7a0_bae6_45de_ba06_36ff04fdbcb4.slice/crio-c0522ed12f516236d9377604f043f6736addc3258a17d44d19446ca2acc76332 WatchSource:0}: Error finding container c0522ed12f516236d9377604f043f6736addc3258a17d44d19446ca2acc76332: Status 404 returned error can't find the container with id c0522ed12f516236d9377604f043f6736addc3258a17d44d19446ca2acc76332 Sep 15 21:13:45.221022 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:45.220993 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:13:45.221092 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:45.221058 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:13:45.221134 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:45.221089 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:13:45.353296 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:45.353248 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" event={"ID":"c101f7a0-bae6-45de-ba06-36ff04fdbcb4","Type":"ContainerStarted","Data":"980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb"} Sep 15 21:13:45.353460 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:45.353330 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" event={"ID":"c101f7a0-bae6-45de-ba06-36ff04fdbcb4","Type":"ContainerStarted","Data":"c0522ed12f516236d9377604f043f6736addc3258a17d44d19446ca2acc76332"} Sep 15 21:13:45.373745 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:45.373693 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" podStartSLOduration=1.373674461 podStartE2EDuration="1.373674461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:45.370747712 +0000 UTC m=+2222.564701180" watchObservedRunningTime="2026-09-15 21:13:45.373674461 +0000 UTC m=+2222.567627916" Sep 15 21:13:45.937648 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:45.937613 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:46.943762 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:46.943731 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:47.364353 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:47.364323 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:13:47.365330 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:13:47.365311 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:14:07.467914 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:07.467881 2579 generic.go:356] "Generic (PLEG): container finished" podID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerID="980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb" exitCode=0 Sep 15 21:14:07.468360 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:07.467962 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" event={"ID":"c101f7a0-bae6-45de-ba06-36ff04fdbcb4","Type":"ContainerDied","Data":"980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb"} Sep 15 21:14:07.468413 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:07.468385 2579 scope.go:117] "RemoveContainer" containerID="980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb" Sep 15 21:14:07.469612 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:07.469583 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:14:07.469731 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:07.469667 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:14:07.469731 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:07.469717 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 15 21:14:08.475795 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:08.475754 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" event={"ID":"c101f7a0-bae6-45de-ba06-36ff04fdbcb4","Type":"ContainerStarted","Data":"67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9"} Sep 15 21:14:08.938411 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:08.938370 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:14:08.938681 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:08.938656 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Sep 15 21:14:08.938737 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:08.938713 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Sep 15 21:14:09.938294 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:09.938253 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Sep 15 21:14:09.938733 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:09.938314 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Sep 15 21:14:10.938482 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:10.938447 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Sep 15 21:14:10.938904 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:10.938523 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Sep 15 21:14:11.938648 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:11.938609 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Sep 15 21:14:11.939068 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:11.938668 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Sep 15 21:14:12.938134 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:12.938095 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Sep 15 21:14:12.938326 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:12.938157 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Sep 15 21:14:12.981994 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:12.981950 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m"] Sep 15 21:14:12.986247 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:12.986219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m" Sep 15 21:14:12.994242 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:12.994200 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m"] Sep 15 21:14:12.994562 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:12.994536 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:14:13.042789 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:13.042759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab04ae28-55dd-4212-821b-d956e5bc5b89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m\" (UID: \"ab04ae28-55dd-4212-821b-d956e5bc5b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m" Sep 15 21:14:13.143340 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:13.143303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab04ae28-55dd-4212-821b-d956e5bc5b89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m\" (UID: \"ab04ae28-55dd-4212-821b-d956e5bc5b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m" Sep 15 21:14:13.143947 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:13.143928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab04ae28-55dd-4212-821b-d956e5bc5b89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m\" (UID: \"ab04ae28-55dd-4212-821b-d956e5bc5b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m" Sep 15 21:14:13.308130 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:13.308090 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m" Sep 15 21:14:13.583150 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:13.583125 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m"] Sep 15 21:14:13.586115 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:14:13.586084 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab04ae28_55dd_4212_821b_d956e5bc5b89.slice/crio-38f224b5c7ee6b10f01f5720efac97aec20b8c68a6846bd68100f858bcc98c9f WatchSource:0}: Error finding container 38f224b5c7ee6b10f01f5720efac97aec20b8c68a6846bd68100f858bcc98c9f: Status 404 returned error can't find the container with id 38f224b5c7ee6b10f01f5720efac97aec20b8c68a6846bd68100f858bcc98c9f Sep 15 21:14:13.942738 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:13.942701 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:14:13.943085 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:13.943057 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:14:13.943661 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:13.943640 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:14:14.520136 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:14.520101 2579 generic.go:356] "Generic (PLEG): container finished" podID="ab04ae28-55dd-4212-821b-d956e5bc5b89" containerID="d1aeb51d7917c56261162efc5882de556fc09339d2aa718f35d8cc4ee3d2310f" exitCode=0 Sep 15 21:14:14.520554 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:14.520182 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m" event={"ID":"ab04ae28-55dd-4212-821b-d956e5bc5b89","Type":"ContainerDied","Data":"d1aeb51d7917c56261162efc5882de556fc09339d2aa718f35d8cc4ee3d2310f"} Sep 15 21:14:14.520554 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:14.520234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m" event={"ID":"ab04ae28-55dd-4212-821b-d956e5bc5b89","Type":"ContainerStarted","Data":"38f224b5c7ee6b10f01f5720efac97aec20b8c68a6846bd68100f858bcc98c9f"} Sep 15 21:14:15.660752 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:15.660726 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m" Sep 15 21:14:15.767707 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:15.767672 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab04ae28-55dd-4212-821b-d956e5bc5b89-maas-api-service-ca\") pod \"ab04ae28-55dd-4212-821b-d956e5bc5b89\" (UID: \"ab04ae28-55dd-4212-821b-d956e5bc5b89\") " Sep 15 21:14:15.768083 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:15.768061 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab04ae28-55dd-4212-821b-d956e5bc5b89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab04ae28-55dd-4212-821b-d956e5bc5b89" (UID: "ab04ae28-55dd-4212-821b-d956e5bc5b89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:14:15.868953 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:15.868876 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab04ae28-55dd-4212-821b-d956e5bc5b89-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:16.531926 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:16.531894 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m" Sep 15 21:14:16.531926 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:16.531917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m" event={"ID":"ab04ae28-55dd-4212-821b-d956e5bc5b89","Type":"ContainerDied","Data":"38f224b5c7ee6b10f01f5720efac97aec20b8c68a6846bd68100f858bcc98c9f"} Sep 15 21:14:16.532154 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:16.531967 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38f224b5c7ee6b10f01f5720efac97aec20b8c68a6846bd68100f858bcc98c9f" Sep 15 21:14:19.576531 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:19.576465 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m"] Sep 15 21:14:19.579053 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:19.579028 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a6f49f-ztx7m"] Sep 15 21:14:21.312004 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:21.311970 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab04ae28-55dd-4212-821b-d956e5bc5b89" path="/var/lib/kubelet/pods/ab04ae28-55dd-4212-821b-d956e5bc5b89/volumes" Sep 15 21:14:26.027946 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:26.027913 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd"] Sep 15 21:14:26.028370 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:26.028158 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" containerID="cri-o://67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9" gracePeriod=30 Sep 15 21:14:28.945702 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:28.945661 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 15 21:14:28.946125 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:28.945755 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:14:29.672033 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.671996 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8"] Sep 15 21:14:29.672439 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.672425 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab04ae28-55dd-4212-821b-d956e5bc5b89" containerName="revoke-keys" Sep 15 21:14:29.672491 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.672442 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab04ae28-55dd-4212-821b-d956e5bc5b89" containerName="revoke-keys" Sep 15 21:14:29.672545 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.672537 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab04ae28-55dd-4212-821b-d956e5bc5b89" containerName="revoke-keys" Sep 15 21:14:29.675734 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.675718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" Sep 15 21:14:29.677967 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.677946 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:14:29.683678 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.683611 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8"] Sep 15 21:14:29.799422 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.799379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c9d86e-5c1d-4ab3-a28e-a0ae06e70871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8\" (UID: \"70c9d86e-5c1d-4ab3-a28e-a0ae06e70871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" Sep 15 21:14:29.900394 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.900356 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c9d86e-5c1d-4ab3-a28e-a0ae06e70871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8\" (UID: \"70c9d86e-5c1d-4ab3-a28e-a0ae06e70871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" Sep 15 21:14:29.900989 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.900967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c9d86e-5c1d-4ab3-a28e-a0ae06e70871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8\" (UID: \"70c9d86e-5c1d-4ab3-a28e-a0ae06e70871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" Sep 15 21:14:29.996764 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:29.996690 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" Sep 15 21:14:30.270315 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:30.270287 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8"] Sep 15 21:14:30.272441 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:14:30.272409 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c9d86e_5c1d_4ab3_a28e_a0ae06e70871.slice/crio-381e12ceb9b72fdfaf4bf16def4130774b81d86c09b5dc2d8db43f00bca762de WatchSource:0}: Error finding container 381e12ceb9b72fdfaf4bf16def4130774b81d86c09b5dc2d8db43f00bca762de: Status 404 returned error can't find the container with id 381e12ceb9b72fdfaf4bf16def4130774b81d86c09b5dc2d8db43f00bca762de Sep 15 21:14:30.610754 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:30.610711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" event={"ID":"70c9d86e-5c1d-4ab3-a28e-a0ae06e70871","Type":"ContainerStarted","Data":"603e23c1da15b13d1bff075e35e15876483d4a9f537cff59541c98ab3a32dc97"} Sep 15 21:14:30.610919 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:30.610767 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" event={"ID":"70c9d86e-5c1d-4ab3-a28e-a0ae06e70871","Type":"ContainerStarted","Data":"381e12ceb9b72fdfaf4bf16def4130774b81d86c09b5dc2d8db43f00bca762de"} Sep 15 21:14:30.628911 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:30.628858 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" podStartSLOduration=1.6288399409999998 podStartE2EDuration="1.628839941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:30.627027706 +0000 UTC m=+2267.820981162" watchObservedRunningTime="2026-09-15 21:14:30.628839941 +0000 UTC m=+2267.822793395" Sep 15 21:14:31.276052 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.276030 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:14:31.415237 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415204 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istiod-ca-cert\") pod \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " Sep 15 21:14:31.415442 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415257 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-certs\") pod \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " Sep 15 21:14:31.415442 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415299 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-data\") pod \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " Sep 15 21:14:31.415442 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415405 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-podinfo\") pod \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " Sep 15 21:14:31.415652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415522 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-token\") pod \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " Sep 15 21:14:31.415652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415561 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-socket\") pod \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " Sep 15 21:14:31.415652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415606 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnz9z\" (UniqueName: \"kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-kube-api-access-pnz9z\") pod \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " Sep 15 21:14:31.415652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415618 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "c101f7a0-bae6-45de-ba06-36ff04fdbcb4" (UID: "c101f7a0-bae6-45de-ba06-36ff04fdbcb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:14:31.415652 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415641 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c101f7a0-bae6-45de-ba06-36ff04fdbcb4" (UID: "c101f7a0-bae6-45de-ba06-36ff04fdbcb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:14:31.415920 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415668 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-envoy\") pod \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " Sep 15 21:14:31.415920 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415675 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c101f7a0-bae6-45de-ba06-36ff04fdbcb4" (UID: "c101f7a0-bae6-45de-ba06-36ff04fdbcb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:14:31.415920 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415713 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-credential-socket\") pod \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\" (UID: \"c101f7a0-bae6-45de-ba06-36ff04fdbcb4\") " Sep 15 21:14:31.415920 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.415837 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c101f7a0-bae6-45de-ba06-36ff04fdbcb4" (UID: "c101f7a0-bae6-45de-ba06-36ff04fdbcb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:14:31.416146 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.416020 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c101f7a0-bae6-45de-ba06-36ff04fdbcb4" (UID: "c101f7a0-bae6-45de-ba06-36ff04fdbcb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:14:31.416146 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.416134 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:31.416256 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.416153 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:31.416256 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.416167 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:31.416256 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.416179 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:31.416256 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.416190 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:31.417947 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.417915 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "c101f7a0-bae6-45de-ba06-36ff04fdbcb4" (UID: "c101f7a0-bae6-45de-ba06-36ff04fdbcb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:14:31.418052 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.417969 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c101f7a0-bae6-45de-ba06-36ff04fdbcb4" (UID: "c101f7a0-bae6-45de-ba06-36ff04fdbcb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:14:31.418281 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.418252 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c101f7a0-bae6-45de-ba06-36ff04fdbcb4" (UID: "c101f7a0-bae6-45de-ba06-36ff04fdbcb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:14:31.418281 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.418261 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-kube-api-access-pnz9z" (OuterVolumeSpecName: "kube-api-access-pnz9z") pod "c101f7a0-bae6-45de-ba06-36ff04fdbcb4" (UID: "c101f7a0-bae6-45de-ba06-36ff04fdbcb4"). InnerVolumeSpecName "kube-api-access-pnz9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:14:31.517120 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.517088 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:31.517120 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.517119 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:31.517323 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.517128 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:31.517323 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.517139 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnz9z\" (UniqueName: \"kubernetes.io/projected/c101f7a0-bae6-45de-ba06-36ff04fdbcb4-kube-api-access-pnz9z\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:31.617989 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.617952 2579 generic.go:356] "Generic (PLEG): container finished" podID="70c9d86e-5c1d-4ab3-a28e-a0ae06e70871" containerID="603e23c1da15b13d1bff075e35e15876483d4a9f537cff59541c98ab3a32dc97" exitCode=0 Sep 15 21:14:31.618171 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.618038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" event={"ID":"70c9d86e-5c1d-4ab3-a28e-a0ae06e70871","Type":"ContainerDied","Data":"603e23c1da15b13d1bff075e35e15876483d4a9f537cff59541c98ab3a32dc97"} Sep 15 21:14:31.619921 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.619898 2579 generic.go:356] "Generic (PLEG): container finished" podID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerID="67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9" exitCode=0 Sep 15 21:14:31.620056 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.619937 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" event={"ID":"c101f7a0-bae6-45de-ba06-36ff04fdbcb4","Type":"ContainerDied","Data":"67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9"} Sep 15 21:14:31.620056 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.619947 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" Sep 15 21:14:31.620056 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.619982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd" event={"ID":"c101f7a0-bae6-45de-ba06-36ff04fdbcb4","Type":"ContainerDied","Data":"c0522ed12f516236d9377604f043f6736addc3258a17d44d19446ca2acc76332"} Sep 15 21:14:31.620056 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.620012 2579 scope.go:117] "RemoveContainer" containerID="67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9" Sep 15 21:14:31.629781 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.629762 2579 scope.go:117] "RemoveContainer" containerID="980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb" Sep 15 21:14:31.638476 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.638456 2579 scope.go:117] "RemoveContainer" containerID="67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9" Sep 15 21:14:31.638780 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:14:31.638760 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9\": container with ID starting with 67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9 not found: ID does not exist" containerID="67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9" Sep 15 21:14:31.638836 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.638790 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9"} err="failed to get container status \"67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9\": rpc error: code = NotFound desc = could not find container \"67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9\": container with ID starting with 67193fd220c8837660287dc858631d7734188eaf049fa540e17c0b4bc99f3ce9 not found: ID does not exist" Sep 15 21:14:31.638836 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.638809 2579 scope.go:117] "RemoveContainer" containerID="980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb" Sep 15 21:14:31.639046 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:14:31.639029 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb\": container with ID starting with 980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb not found: ID does not exist" containerID="980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb" Sep 15 21:14:31.639089 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.639052 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb"} err="failed to get container status \"980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb\": rpc error: code = NotFound desc = could not find container \"980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb\": container with ID starting with 980519b7643f7a2f3a003ba293d3509ef5d37f8614cc9f64395b52be921f7dfb not found: ID does not exist" Sep 15 21:14:31.672958 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.672919 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd"] Sep 15 21:14:31.678970 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:31.678937 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a6f49f-openshift-default-97f555584-sk4pd"] Sep 15 21:14:32.762342 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:32.762316 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" Sep 15 21:14:32.829383 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:32.829348 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c9d86e-5c1d-4ab3-a28e-a0ae06e70871-maas-api-service-ca\") pod \"70c9d86e-5c1d-4ab3-a28e-a0ae06e70871\" (UID: \"70c9d86e-5c1d-4ab3-a28e-a0ae06e70871\") " Sep 15 21:14:32.829776 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:32.829752 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c9d86e-5c1d-4ab3-a28e-a0ae06e70871-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70c9d86e-5c1d-4ab3-a28e-a0ae06e70871" (UID: "70c9d86e-5c1d-4ab3-a28e-a0ae06e70871"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:14:32.930432 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:32.930350 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c9d86e-5c1d-4ab3-a28e-a0ae06e70871-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:14:33.312605 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:33.312564 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" path="/var/lib/kubelet/pods/c101f7a0-bae6-45de-ba06-36ff04fdbcb4/volumes" Sep 15 21:14:33.635527 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:33.635421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" event={"ID":"70c9d86e-5c1d-4ab3-a28e-a0ae06e70871","Type":"ContainerDied","Data":"381e12ceb9b72fdfaf4bf16def4130774b81d86c09b5dc2d8db43f00bca762de"} Sep 15 21:14:33.635527 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:33.635469 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="381e12ceb9b72fdfaf4bf16def4130774b81d86c09b5dc2d8db43f00bca762de" Sep 15 21:14:33.635527 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:33.635432 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8" Sep 15 21:14:43.893876 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:43.893845 2579 scope.go:117] "RemoveContainer" containerID="8b3f1a230b6b85e44d34c5c11cbaf0f70ee79430cb1cbec43df5bb5d1ddb10cc" Sep 15 21:14:43.903734 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:43.903701 2579 scope.go:117] "RemoveContainer" containerID="9c6be14c070de37cdf8398b73f6f391872199afdc434b5decc97e0d09fc69dee" Sep 15 21:14:43.912393 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:43.912371 2579 scope.go:117] "RemoveContainer" containerID="d955ec0d7c4fdf5f35115212dfdb22fb934e02eb6dd27196a696113f484e3c5c" Sep 15 21:14:45.434455 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:45.434400 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8"] Sep 15 21:14:45.436626 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:45.436604 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219de8-5pzv8"] Sep 15 21:14:47.311542 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:14:47.311481 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c9d86e-5c1d-4ab3-a28e-a0ae06e70871" path="/var/lib/kubelet/pods/70c9d86e-5c1d-4ab3-a28e-a0ae06e70871/volumes" Sep 15 21:15:00.136325 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.136291 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2"] Sep 15 21:15:00.136816 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.136725 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" Sep 15 21:15:00.136816 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.136737 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" Sep 15 21:15:00.136816 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.136747 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" Sep 15 21:15:00.136816 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.136752 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" Sep 15 21:15:00.136816 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.136761 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c9d86e-5c1d-4ab3-a28e-a0ae06e70871" containerName="revoke-keys" Sep 15 21:15:00.136816 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.136767 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c9d86e-5c1d-4ab3-a28e-a0ae06e70871" containerName="revoke-keys" Sep 15 21:15:00.137014 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.136848 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c9d86e-5c1d-4ab3-a28e-a0ae06e70871" containerName="revoke-keys" Sep 15 21:15:00.137014 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.136859 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" Sep 15 21:15:00.137014 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.136867 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c101f7a0-bae6-45de-ba06-36ff04fdbcb4" containerName="istio-proxy" Sep 15 21:15:00.140063 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.140041 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5"] Sep 15 21:15:00.140192 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.140152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2" Sep 15 21:15:00.142276 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.142248 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:15:00.143314 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.143300 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5" Sep 15 21:15:00.148348 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.148324 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2"] Sep 15 21:15:00.152870 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.152849 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5"] Sep 15 21:15:00.153891 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.153875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2" Sep 15 21:15:00.162434 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.162415 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5" Sep 15 21:15:00.429829 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.429803 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2"] Sep 15 21:15:00.432204 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:15:00.432176 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d4f7a1_61ec_4a22_822a_77830205301e.slice/crio-baef3a2587b81109139e753877ec706e289e71889aefbcf7d2572eb697a1cbae WatchSource:0}: Error finding container baef3a2587b81109139e753877ec706e289e71889aefbcf7d2572eb697a1cbae: Status 404 returned error can't find the container with id baef3a2587b81109139e753877ec706e289e71889aefbcf7d2572eb697a1cbae Sep 15 21:15:00.451709 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.451686 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5"] Sep 15 21:15:00.454209 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:15:00.454182 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9423a8_43bc_4b51_844b_766247f2cc44.slice/crio-28a71226667000baafa3de8a54de72f04668ddd1912d88ee8a5f79c1a7351409 WatchSource:0}: Error finding container 28a71226667000baafa3de8a54de72f04668ddd1912d88ee8a5f79c1a7351409: Status 404 returned error can't find the container with id 28a71226667000baafa3de8a54de72f04668ddd1912d88ee8a5f79c1a7351409 Sep 15 21:15:00.784488 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.784460 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5" event={"ID":"3a9423a8-43bc-4b51-844b-766247f2cc44","Type":"ContainerStarted","Data":"28a71226667000baafa3de8a54de72f04668ddd1912d88ee8a5f79c1a7351409"} Sep 15 21:15:00.786462 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.786429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2" event={"ID":"12d4f7a1-61ec-4a22-822a-77830205301e","Type":"ContainerStarted","Data":"0b1366e47c74ed4ce0ce36de0dd18661b1617fd74f86a5ee401d60ec7677e2c0"} Sep 15 21:15:00.786629 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.786489 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2" event={"ID":"12d4f7a1-61ec-4a22-822a-77830205301e","Type":"ContainerStarted","Data":"baef3a2587b81109139e753877ec706e289e71889aefbcf7d2572eb697a1cbae"} Sep 15 21:15:00.801021 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:00.800954 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2" podStartSLOduration=0.800938063 podStartE2EDuration="800.938063ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:00.800191954 +0000 UTC m=+2297.994145411" watchObservedRunningTime="2026-09-15 21:15:00.800938063 +0000 UTC m=+2297.994891515" Sep 15 21:15:01.794012 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:01.793976 2579 generic.go:356] "Generic (PLEG): container finished" podID="12d4f7a1-61ec-4a22-822a-77830205301e" containerID="0b1366e47c74ed4ce0ce36de0dd18661b1617fd74f86a5ee401d60ec7677e2c0" exitCode=0 Sep 15 21:15:01.794449 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:01.794062 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2" event={"ID":"12d4f7a1-61ec-4a22-822a-77830205301e","Type":"ContainerDied","Data":"0b1366e47c74ed4ce0ce36de0dd18661b1617fd74f86a5ee401d60ec7677e2c0"} Sep 15 21:15:01.795715 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:01.795691 2579 generic.go:356] "Generic (PLEG): container finished" podID="3a9423a8-43bc-4b51-844b-766247f2cc44" containerID="78d9eca09c05b51976bd65c872103451fbf1b36f99006a75c3873e0c490e44ef" exitCode=0 Sep 15 21:15:01.795838 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:01.795768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5" event={"ID":"3a9423a8-43bc-4b51-844b-766247f2cc44","Type":"ContainerDied","Data":"78d9eca09c05b51976bd65c872103451fbf1b36f99006a75c3873e0c490e44ef"} Sep 15 21:15:02.960039 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:02.960012 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2" Sep 15 21:15:02.963055 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:02.963033 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5" Sep 15 21:15:03.807242 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:03.807212 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5" Sep 15 21:15:03.807433 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:03.807262 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5" event={"ID":"3a9423a8-43bc-4b51-844b-766247f2cc44","Type":"ContainerDied","Data":"28a71226667000baafa3de8a54de72f04668ddd1912d88ee8a5f79c1a7351409"} Sep 15 21:15:03.807433 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:03.807312 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28a71226667000baafa3de8a54de72f04668ddd1912d88ee8a5f79c1a7351409" Sep 15 21:15:03.808606 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:03.808584 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2" Sep 15 21:15:03.808606 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:03.808590 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825115-2j9p2" event={"ID":"12d4f7a1-61ec-4a22-822a-77830205301e","Type":"ContainerDied","Data":"baef3a2587b81109139e753877ec706e289e71889aefbcf7d2572eb697a1cbae"} Sep 15 21:15:03.808808 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:03.808630 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baef3a2587b81109139e753877ec706e289e71889aefbcf7d2572eb697a1cbae" Sep 15 21:15:34.263509 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.263461 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs"] Sep 15 21:15:34.263978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.263918 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9423a8-43bc-4b51-844b-766247f2cc44" containerName="cleanup" Sep 15 21:15:34.263978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.263931 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9423a8-43bc-4b51-844b-766247f2cc44" containerName="cleanup" Sep 15 21:15:34.263978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.263940 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d4f7a1-61ec-4a22-822a-77830205301e" containerName="cleanup" Sep 15 21:15:34.263978 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.263946 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d4f7a1-61ec-4a22-822a-77830205301e" containerName="cleanup" Sep 15 21:15:34.264118 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.264009 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="12d4f7a1-61ec-4a22-822a-77830205301e" containerName="cleanup" Sep 15 21:15:34.264118 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.264020 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a9423a8-43bc-4b51-844b-766247f2cc44" containerName="cleanup" Sep 15 21:15:34.267338 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.267315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs" Sep 15 21:15:34.270754 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.270735 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:15:34.283606 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.283575 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs"] Sep 15 21:15:34.322700 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.322665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbeb98bf-623a-42f8-9104-e8ffe0c91aab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs\" (UID: \"bbeb98bf-623a-42f8-9104-e8ffe0c91aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs" Sep 15 21:15:34.423335 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.423241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbeb98bf-623a-42f8-9104-e8ffe0c91aab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs\" (UID: \"bbeb98bf-623a-42f8-9104-e8ffe0c91aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs" Sep 15 21:15:34.423898 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.423878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbeb98bf-623a-42f8-9104-e8ffe0c91aab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs\" (UID: \"bbeb98bf-623a-42f8-9104-e8ffe0c91aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs" Sep 15 21:15:34.578958 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.578871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs" Sep 15 21:15:34.850243 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.850219 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs"] Sep 15 21:15:34.852535 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:15:34.852483 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbeb98bf_623a_42f8_9104_e8ffe0c91aab.slice/crio-75ed76be3acd88283abd2cc327f4131727f6426fd169b6a8e769f07d82a8bd3c WatchSource:0}: Error finding container 75ed76be3acd88283abd2cc327f4131727f6426fd169b6a8e769f07d82a8bd3c: Status 404 returned error can't find the container with id 75ed76be3acd88283abd2cc327f4131727f6426fd169b6a8e769f07d82a8bd3c Sep 15 21:15:34.968591 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:34.968554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs" event={"ID":"bbeb98bf-623a-42f8-9104-e8ffe0c91aab","Type":"ContainerStarted","Data":"75ed76be3acd88283abd2cc327f4131727f6426fd169b6a8e769f07d82a8bd3c"} Sep 15 21:15:35.975418 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:35.975320 2579 generic.go:356] "Generic (PLEG): container finished" podID="bbeb98bf-623a-42f8-9104-e8ffe0c91aab" containerID="371ad03319aa21df9e37e6b5406a914949d5602983d60dc5c000a7cab1d1e0a8" exitCode=0 Sep 15 21:15:35.975418 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:35.975397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs" event={"ID":"bbeb98bf-623a-42f8-9104-e8ffe0c91aab","Type":"ContainerDied","Data":"371ad03319aa21df9e37e6b5406a914949d5602983d60dc5c000a7cab1d1e0a8"} Sep 15 21:15:37.113371 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:37.113347 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs" Sep 15 21:15:37.148691 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:37.148656 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbeb98bf-623a-42f8-9104-e8ffe0c91aab-maas-api-service-ca\") pod \"bbeb98bf-623a-42f8-9104-e8ffe0c91aab\" (UID: \"bbeb98bf-623a-42f8-9104-e8ffe0c91aab\") " Sep 15 21:15:37.149041 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:37.149015 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbeb98bf-623a-42f8-9104-e8ffe0c91aab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bbeb98bf-623a-42f8-9104-e8ffe0c91aab" (UID: "bbeb98bf-623a-42f8-9104-e8ffe0c91aab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:15:37.250153 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:37.250069 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbeb98bf-623a-42f8-9104-e8ffe0c91aab-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:37.987363 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:37.987325 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs" Sep 15 21:15:37.987577 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:37.987361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs" event={"ID":"bbeb98bf-623a-42f8-9104-e8ffe0c91aab","Type":"ContainerDied","Data":"75ed76be3acd88283abd2cc327f4131727f6426fd169b6a8e769f07d82a8bd3c"} Sep 15 21:15:37.987577 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:37.987436 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ed76be3acd88283abd2cc327f4131727f6426fd169b6a8e769f07d82a8bd3c" Sep 15 21:15:43.948530 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:43.948474 2579 scope.go:117] "RemoveContainer" containerID="d63f4f653c6fe2231e48ab9a9b67c0a317c14d26a212c6bf0ff06a6d182b269b" Sep 15 21:15:43.961291 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:43.961263 2579 scope.go:117] "RemoveContainer" containerID="1da090bf17f1e18db7a6cd2439e91dd28a7da9d02ff1a46d9bc65c6335ec8e17" Sep 15 21:15:44.929833 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:44.929798 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2"] Sep 15 21:15:44.930154 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:44.930068 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" podUID="ec362713-c4d6-4cf1-b223-62c15e950f84" containerName="payload-processing" containerID="cri-o://c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55" gracePeriod=30 Sep 15 21:15:44.937007 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:44.936977 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw"] Sep 15 21:15:44.937791 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:44.937758 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" podUID="93108ba3-b5da-4f39-bdc1-70c2ff12f454" containerName="payload-pre-processing" containerID="cri-o://8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421" gracePeriod=30 Sep 15 21:15:45.149269 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.149200 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs"] Sep 15 21:15:45.151809 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.151785 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f09a7-25pfs"] Sep 15 21:15:45.217527 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.217481 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:15:45.220109 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.220090 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:15:45.311779 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.311749 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbeb98bf-623a-42f8-9104-e8ffe0c91aab" path="/var/lib/kubelet/pods/bbeb98bf-623a-42f8-9104-e8ffe0c91aab/volumes" Sep 15 21:15:45.329208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.329183 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsvm2\" (UniqueName: \"kubernetes.io/projected/93108ba3-b5da-4f39-bdc1-70c2ff12f454-kube-api-access-nsvm2\") pod \"93108ba3-b5da-4f39-bdc1-70c2ff12f454\" (UID: \"93108ba3-b5da-4f39-bdc1-70c2ff12f454\") " Sep 15 21:15:45.329365 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.329218 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec362713-c4d6-4cf1-b223-62c15e950f84-plugins-config-volume\") pod \"ec362713-c4d6-4cf1-b223-62c15e950f84\" (UID: \"ec362713-c4d6-4cf1-b223-62c15e950f84\") " Sep 15 21:15:45.329365 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.329323 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcbpz\" (UniqueName: \"kubernetes.io/projected/ec362713-c4d6-4cf1-b223-62c15e950f84-kube-api-access-zcbpz\") pod \"ec362713-c4d6-4cf1-b223-62c15e950f84\" (UID: \"ec362713-c4d6-4cf1-b223-62c15e950f84\") " Sep 15 21:15:45.329365 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.329356 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93108ba3-b5da-4f39-bdc1-70c2ff12f454-plugins-config-volume\") pod \"93108ba3-b5da-4f39-bdc1-70c2ff12f454\" (UID: \"93108ba3-b5da-4f39-bdc1-70c2ff12f454\") " Sep 15 21:15:45.329723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.329695 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec362713-c4d6-4cf1-b223-62c15e950f84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec362713-c4d6-4cf1-b223-62c15e950f84" (UID: "ec362713-c4d6-4cf1-b223-62c15e950f84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:15:45.329826 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.329801 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93108ba3-b5da-4f39-bdc1-70c2ff12f454-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93108ba3-b5da-4f39-bdc1-70c2ff12f454" (UID: "93108ba3-b5da-4f39-bdc1-70c2ff12f454"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:15:45.331413 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.331390 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93108ba3-b5da-4f39-bdc1-70c2ff12f454-kube-api-access-nsvm2" (OuterVolumeSpecName: "kube-api-access-nsvm2") pod "93108ba3-b5da-4f39-bdc1-70c2ff12f454" (UID: "93108ba3-b5da-4f39-bdc1-70c2ff12f454"). InnerVolumeSpecName "kube-api-access-nsvm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:15:45.331677 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.331654 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec362713-c4d6-4cf1-b223-62c15e950f84-kube-api-access-zcbpz" (OuterVolumeSpecName: "kube-api-access-zcbpz") pod "ec362713-c4d6-4cf1-b223-62c15e950f84" (UID: "ec362713-c4d6-4cf1-b223-62c15e950f84"). InnerVolumeSpecName "kube-api-access-zcbpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:15:45.430870 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.430831 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcbpz\" (UniqueName: \"kubernetes.io/projected/ec362713-c4d6-4cf1-b223-62c15e950f84-kube-api-access-zcbpz\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:45.430870 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.430863 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93108ba3-b5da-4f39-bdc1-70c2ff12f454-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:45.430870 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.430872 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsvm2\" (UniqueName: \"kubernetes.io/projected/93108ba3-b5da-4f39-bdc1-70c2ff12f454-kube-api-access-nsvm2\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:45.430870 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:45.430882 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec362713-c4d6-4cf1-b223-62c15e950f84-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:46.031926 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.031890 2579 generic.go:356] "Generic (PLEG): container finished" podID="ec362713-c4d6-4cf1-b223-62c15e950f84" containerID="c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55" exitCode=0 Sep 15 21:15:46.032124 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.031934 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" Sep 15 21:15:46.032124 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.031984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" event={"ID":"ec362713-c4d6-4cf1-b223-62c15e950f84","Type":"ContainerDied","Data":"c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55"} Sep 15 21:15:46.032124 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.032038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2" event={"ID":"ec362713-c4d6-4cf1-b223-62c15e950f84","Type":"ContainerDied","Data":"19fbb46515e979a204fa75397ee617527cd89b62dad5e5b514be04d12cb98dc1"} Sep 15 21:15:46.032124 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.032068 2579 scope.go:117] "RemoveContainer" containerID="c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55" Sep 15 21:15:46.033600 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.033576 2579 generic.go:356] "Generic (PLEG): container finished" podID="93108ba3-b5da-4f39-bdc1-70c2ff12f454" containerID="8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421" exitCode=0 Sep 15 21:15:46.033716 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.033644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" event={"ID":"93108ba3-b5da-4f39-bdc1-70c2ff12f454","Type":"ContainerDied","Data":"8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421"} Sep 15 21:15:46.033716 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.033679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" event={"ID":"93108ba3-b5da-4f39-bdc1-70c2ff12f454","Type":"ContainerDied","Data":"1796b29b0de83603a8b90375023f8c0eaf7d52430d69ca8b88bb830fd38894b4"} Sep 15 21:15:46.033716 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.033647 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw" Sep 15 21:15:46.042525 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.042483 2579 scope.go:117] "RemoveContainer" containerID="c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55" Sep 15 21:15:46.042809 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:15:46.042789 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55\": container with ID starting with c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55 not found: ID does not exist" containerID="c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55" Sep 15 21:15:46.042897 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.042819 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55"} err="failed to get container status \"c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55\": rpc error: code = NotFound desc = could not find container \"c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55\": container with ID starting with c8d62a01cde9a7e95cc4658032f2e6221c0a49592903a94d7e91a257c420ef55 not found: ID does not exist" Sep 15 21:15:46.042897 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.042855 2579 scope.go:117] "RemoveContainer" containerID="8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421" Sep 15 21:15:46.051233 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.051215 2579 scope.go:117] "RemoveContainer" containerID="8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421" Sep 15 21:15:46.051510 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:15:46.051476 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421\": container with ID starting with 8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421 not found: ID does not exist" containerID="8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421" Sep 15 21:15:46.051560 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.051522 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421"} err="failed to get container status \"8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421\": rpc error: code = NotFound desc = could not find container \"8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421\": container with ID starting with 8ca52668fb046a6bd5ea805237fd2cfb0d383f06f68f3bb2307da789ba4ea421 not found: ID does not exist" Sep 15 21:15:46.056272 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.056252 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2"] Sep 15 21:15:46.060237 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.060217 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f09a7-d5f867bb7-rc8l2"] Sep 15 21:15:46.071218 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.071195 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw"] Sep 15 21:15:46.073729 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:46.073711 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f09a7-799f644c8b-xtzlw"] Sep 15 21:15:47.311894 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:47.311863 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93108ba3-b5da-4f39-bdc1-70c2ff12f454" path="/var/lib/kubelet/pods/93108ba3-b5da-4f39-bdc1-70c2ff12f454/volumes" Sep 15 21:15:47.312360 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:47.312341 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec362713-c4d6-4cf1-b223-62c15e950f84" path="/var/lib/kubelet/pods/ec362713-c4d6-4cf1-b223-62c15e950f84/volumes" Sep 15 21:15:52.109303 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.109261 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8"] Sep 15 21:15:52.109739 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.109487 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" containerID="cri-o://39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af" gracePeriod=30 Sep 15 21:15:52.480848 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.480763 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt"] Sep 15 21:15:52.481172 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.481158 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93108ba3-b5da-4f39-bdc1-70c2ff12f454" containerName="payload-pre-processing" Sep 15 21:15:52.481220 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.481175 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="93108ba3-b5da-4f39-bdc1-70c2ff12f454" containerName="payload-pre-processing" Sep 15 21:15:52.481220 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.481185 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec362713-c4d6-4cf1-b223-62c15e950f84" containerName="payload-processing" Sep 15 21:15:52.481220 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.481192 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec362713-c4d6-4cf1-b223-62c15e950f84" containerName="payload-processing" Sep 15 21:15:52.481220 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.481216 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbeb98bf-623a-42f8-9104-e8ffe0c91aab" containerName="revoke-keys" Sep 15 21:15:52.481360 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.481223 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbeb98bf-623a-42f8-9104-e8ffe0c91aab" containerName="revoke-keys" Sep 15 21:15:52.481360 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.481288 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec362713-c4d6-4cf1-b223-62c15e950f84" containerName="payload-processing" Sep 15 21:15:52.481360 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.481297 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbeb98bf-623a-42f8-9104-e8ffe0c91aab" containerName="revoke-keys" Sep 15 21:15:52.481360 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.481304 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="93108ba3-b5da-4f39-bdc1-70c2ff12f454" containerName="payload-pre-processing" Sep 15 21:15:52.484374 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.484359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt" Sep 15 21:15:52.486386 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.486365 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:15:52.490113 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.490091 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt"] Sep 15 21:15:52.596725 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.596689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0250e06-0ed2-4ca7-b98c-73b4dc913dc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt\" (UID: \"b0250e06-0ed2-4ca7-b98c-73b4dc913dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt" Sep 15 21:15:52.697238 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.697204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0250e06-0ed2-4ca7-b98c-73b4dc913dc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt\" (UID: \"b0250e06-0ed2-4ca7-b98c-73b4dc913dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt" Sep 15 21:15:52.697850 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.697831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0250e06-0ed2-4ca7-b98c-73b4dc913dc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt\" (UID: \"b0250e06-0ed2-4ca7-b98c-73b4dc913dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt" Sep 15 21:15:52.796717 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:52.796682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt" Sep 15 21:15:53.072631 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:53.072607 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt"] Sep 15 21:15:53.074751 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:15:53.074721 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0250e06_0ed2_4ca7_b98c_73b4dc913dc4.slice/crio-ac6237578fabb8bed1569f82b56a1e5945f3da93e745e4ee9fed89cc48ae1c5f WatchSource:0}: Error finding container ac6237578fabb8bed1569f82b56a1e5945f3da93e745e4ee9fed89cc48ae1c5f: Status 404 returned error can't find the container with id ac6237578fabb8bed1569f82b56a1e5945f3da93e745e4ee9fed89cc48ae1c5f Sep 15 21:15:54.079204 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:54.079171 2579 generic.go:356] "Generic (PLEG): container finished" podID="b0250e06-0ed2-4ca7-b98c-73b4dc913dc4" containerID="e1d3dcd37c4c03b1abe0540d348a97bf879b5e77a9857d7adbbcbae509149389" exitCode=0 Sep 15 21:15:54.079674 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:54.079266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt" event={"ID":"b0250e06-0ed2-4ca7-b98c-73b4dc913dc4","Type":"ContainerDied","Data":"e1d3dcd37c4c03b1abe0540d348a97bf879b5e77a9857d7adbbcbae509149389"} Sep 15 21:15:54.079674 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:54.079329 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt" event={"ID":"b0250e06-0ed2-4ca7-b98c-73b4dc913dc4","Type":"ContainerStarted","Data":"ac6237578fabb8bed1569f82b56a1e5945f3da93e745e4ee9fed89cc48ae1c5f"} Sep 15 21:15:55.222510 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:55.222472 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt" Sep 15 21:15:55.324159 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:55.324130 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0250e06-0ed2-4ca7-b98c-73b4dc913dc4-maas-api-service-ca\") pod \"b0250e06-0ed2-4ca7-b98c-73b4dc913dc4\" (UID: \"b0250e06-0ed2-4ca7-b98c-73b4dc913dc4\") " Sep 15 21:15:55.324551 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:55.324529 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0250e06-0ed2-4ca7-b98c-73b4dc913dc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0250e06-0ed2-4ca7-b98c-73b4dc913dc4" (UID: "b0250e06-0ed2-4ca7-b98c-73b4dc913dc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:15:55.425906 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:55.425807 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0250e06-0ed2-4ca7-b98c-73b4dc913dc4-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:56.090175 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:56.090143 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt" Sep 15 21:15:56.090366 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:56.090184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt" event={"ID":"b0250e06-0ed2-4ca7-b98c-73b4dc913dc4","Type":"ContainerDied","Data":"ac6237578fabb8bed1569f82b56a1e5945f3da93e745e4ee9fed89cc48ae1c5f"} Sep 15 21:15:56.090366 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:56.090233 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac6237578fabb8bed1569f82b56a1e5945f3da93e745e4ee9fed89cc48ae1c5f" Sep 15 21:15:57.359929 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.359900 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:15:57.444826 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.444741 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fggbc\" (UniqueName: \"kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-kube-api-access-fggbc\") pod \"700944ff-55e6-462f-95b0-1222e07aa47f\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " Sep 15 21:15:57.444826 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.444780 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-istio-token\") pod \"700944ff-55e6-462f-95b0-1222e07aa47f\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " Sep 15 21:15:57.444826 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.444816 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700944ff-55e6-462f-95b0-1222e07aa47f-istiod-ca-cert\") pod \"700944ff-55e6-462f-95b0-1222e07aa47f\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " Sep 15 21:15:57.445098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.444857 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-data\") pod \"700944ff-55e6-462f-95b0-1222e07aa47f\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " Sep 15 21:15:57.445098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.444883 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-certs\") pod \"700944ff-55e6-462f-95b0-1222e07aa47f\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " Sep 15 21:15:57.445098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.444912 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-envoy\") pod \"700944ff-55e6-462f-95b0-1222e07aa47f\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " Sep 15 21:15:57.445098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.444959 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-credential-socket\") pod \"700944ff-55e6-462f-95b0-1222e07aa47f\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " Sep 15 21:15:57.445098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.444977 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-socket\") pod \"700944ff-55e6-462f-95b0-1222e07aa47f\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " Sep 15 21:15:57.445098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.445006 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700944ff-55e6-462f-95b0-1222e07aa47f-istio-podinfo\") pod \"700944ff-55e6-462f-95b0-1222e07aa47f\" (UID: \"700944ff-55e6-462f-95b0-1222e07aa47f\") " Sep 15 21:15:57.445419 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.445187 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-data" (OuterVolumeSpecName: "istio-data") pod "700944ff-55e6-462f-95b0-1222e07aa47f" (UID: "700944ff-55e6-462f-95b0-1222e07aa47f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:15:57.445419 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.445217 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "700944ff-55e6-462f-95b0-1222e07aa47f" (UID: "700944ff-55e6-462f-95b0-1222e07aa47f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:15:57.445419 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.445297 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700944ff-55e6-462f-95b0-1222e07aa47f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "700944ff-55e6-462f-95b0-1222e07aa47f" (UID: "700944ff-55e6-462f-95b0-1222e07aa47f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:15:57.445419 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.445359 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:57.445419 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.445378 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:57.445419 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.445412 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "700944ff-55e6-462f-95b0-1222e07aa47f" (UID: "700944ff-55e6-462f-95b0-1222e07aa47f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:15:57.445684 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.445453 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "700944ff-55e6-462f-95b0-1222e07aa47f" (UID: "700944ff-55e6-462f-95b0-1222e07aa47f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:15:57.447202 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.447162 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-istio-token" (OuterVolumeSpecName: "istio-token") pod "700944ff-55e6-462f-95b0-1222e07aa47f" (UID: "700944ff-55e6-462f-95b0-1222e07aa47f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:15:57.447315 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.447249 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-kube-api-access-fggbc" (OuterVolumeSpecName: "kube-api-access-fggbc") pod "700944ff-55e6-462f-95b0-1222e07aa47f" (UID: "700944ff-55e6-462f-95b0-1222e07aa47f"). InnerVolumeSpecName "kube-api-access-fggbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:15:57.447370 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.447354 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/700944ff-55e6-462f-95b0-1222e07aa47f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "700944ff-55e6-462f-95b0-1222e07aa47f" (UID: "700944ff-55e6-462f-95b0-1222e07aa47f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:15:57.447451 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.447437 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "700944ff-55e6-462f-95b0-1222e07aa47f" (UID: "700944ff-55e6-462f-95b0-1222e07aa47f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:15:57.546415 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.546379 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:57.546415 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.546407 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:57.546415 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.546417 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700944ff-55e6-462f-95b0-1222e07aa47f-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:57.546415 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.546426 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700944ff-55e6-462f-95b0-1222e07aa47f-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:57.546699 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.546435 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fggbc\" (UniqueName: \"kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-kube-api-access-fggbc\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:57.546699 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.546444 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700944ff-55e6-462f-95b0-1222e07aa47f-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:57.546699 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:57.546453 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700944ff-55e6-462f-95b0-1222e07aa47f-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:15:58.102134 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.102097 2579 generic.go:356] "Generic (PLEG): container finished" podID="700944ff-55e6-462f-95b0-1222e07aa47f" containerID="39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af" exitCode=0 Sep 15 21:15:58.102324 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.102142 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" Sep 15 21:15:58.102324 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.102177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" event={"ID":"700944ff-55e6-462f-95b0-1222e07aa47f","Type":"ContainerDied","Data":"39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af"} Sep 15 21:15:58.102324 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.102230 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8" event={"ID":"700944ff-55e6-462f-95b0-1222e07aa47f","Type":"ContainerDied","Data":"5d87e51dd052317210253ee5df6d4063f4312c0e0b8da8e35b35e1fcd44a758b"} Sep 15 21:15:58.102324 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.102251 2579 scope.go:117] "RemoveContainer" containerID="39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af" Sep 15 21:15:58.112098 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.112081 2579 scope.go:117] "RemoveContainer" containerID="29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e" Sep 15 21:15:58.120748 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.120726 2579 scope.go:117] "RemoveContainer" containerID="39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af" Sep 15 21:15:58.121091 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:15:58.121070 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af\": container with ID starting with 39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af not found: ID does not exist" containerID="39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af" Sep 15 21:15:58.121206 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.121139 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af"} err="failed to get container status \"39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af\": rpc error: code = NotFound desc = could not find container \"39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af\": container with ID starting with 39a950d410d5a3212adb32008fe52a6ad093d2be99a9b1dc75389e16fe6259af not found: ID does not exist" Sep 15 21:15:58.121206 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.121166 2579 scope.go:117] "RemoveContainer" containerID="29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e" Sep 15 21:15:58.121590 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:15:58.121528 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e\": container with ID starting with 29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e not found: ID does not exist" containerID="29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e" Sep 15 21:15:58.121590 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.121562 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e"} err="failed to get container status \"29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e\": rpc error: code = NotFound desc = could not find container \"29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e\": container with ID starting with 29b1427c75ac8c456566a6d8a8cca16318322b0e18d5acd22f75d83e1e50f39e not found: ID does not exist" Sep 15 21:15:58.123852 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.123833 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8"] Sep 15 21:15:58.127239 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:58.127218 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f09a7-openshift-default-7cffbbf96d-2svw8"] Sep 15 21:15:59.311727 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:15:59.311684 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" path="/var/lib/kubelet/pods/700944ff-55e6-462f-95b0-1222e07aa47f/volumes" Sep 15 21:16:02.989888 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:02.989852 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5"] Sep 15 21:16:02.993594 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:02.993562 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-280fb4-29825115-lz7b5"] Sep 15 21:16:03.015734 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.015704 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5"] Sep 15 21:16:03.015967 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.015941 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" podUID="246fa19f-ba51-465a-818c-4c19d3786857" containerName="payload-processing" containerID="cri-o://ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c" gracePeriod=30 Sep 15 21:16:03.019527 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.019477 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z"] Sep 15 21:16:03.019738 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.019699 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" podUID="c73877a8-533a-4d00-9755-917bee4519d4" containerName="payload-pre-processing" containerID="cri-o://c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0" gracePeriod=30 Sep 15 21:16:03.178070 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.178033 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt"] Sep 15 21:16:03.181602 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.181572 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-280fb4-g5rdt"] Sep 15 21:16:03.274068 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.274038 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:16:03.301973 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.301949 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:16:03.311205 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.311177 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a9423a8-43bc-4b51-844b-766247f2cc44" path="/var/lib/kubelet/pods/3a9423a8-43bc-4b51-844b-766247f2cc44/volumes" Sep 15 21:16:03.311554 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.311539 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0250e06-0ed2-4ca7-b98c-73b4dc913dc4" path="/var/lib/kubelet/pods/b0250e06-0ed2-4ca7-b98c-73b4dc913dc4/volumes" Sep 15 21:16:03.403737 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.403703 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fftwg\" (UniqueName: \"kubernetes.io/projected/c73877a8-533a-4d00-9755-917bee4519d4-kube-api-access-fftwg\") pod \"c73877a8-533a-4d00-9755-917bee4519d4\" (UID: \"c73877a8-533a-4d00-9755-917bee4519d4\") " Sep 15 21:16:03.403737 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.403749 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcqjb\" (UniqueName: \"kubernetes.io/projected/246fa19f-ba51-465a-818c-4c19d3786857-kube-api-access-dcqjb\") pod \"246fa19f-ba51-465a-818c-4c19d3786857\" (UID: \"246fa19f-ba51-465a-818c-4c19d3786857\") " Sep 15 21:16:03.404007 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.403773 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73877a8-533a-4d00-9755-917bee4519d4-plugins-config-volume\") pod \"c73877a8-533a-4d00-9755-917bee4519d4\" (UID: \"c73877a8-533a-4d00-9755-917bee4519d4\") " Sep 15 21:16:03.404007 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.403804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246fa19f-ba51-465a-818c-4c19d3786857-plugins-config-volume\") pod \"246fa19f-ba51-465a-818c-4c19d3786857\" (UID: \"246fa19f-ba51-465a-818c-4c19d3786857\") " Sep 15 21:16:03.404160 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.404133 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73877a8-533a-4d00-9755-917bee4519d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c73877a8-533a-4d00-9755-917bee4519d4" (UID: "c73877a8-533a-4d00-9755-917bee4519d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:16:03.404247 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.404223 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246fa19f-ba51-465a-818c-4c19d3786857-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "246fa19f-ba51-465a-818c-4c19d3786857" (UID: "246fa19f-ba51-465a-818c-4c19d3786857"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:16:03.406125 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.406099 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246fa19f-ba51-465a-818c-4c19d3786857-kube-api-access-dcqjb" (OuterVolumeSpecName: "kube-api-access-dcqjb") pod "246fa19f-ba51-465a-818c-4c19d3786857" (UID: "246fa19f-ba51-465a-818c-4c19d3786857"). InnerVolumeSpecName "kube-api-access-dcqjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:16:03.406235 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.406203 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73877a8-533a-4d00-9755-917bee4519d4-kube-api-access-fftwg" (OuterVolumeSpecName: "kube-api-access-fftwg") pod "c73877a8-533a-4d00-9755-917bee4519d4" (UID: "c73877a8-533a-4d00-9755-917bee4519d4"). InnerVolumeSpecName "kube-api-access-fftwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:16:03.504567 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.504465 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fftwg\" (UniqueName: \"kubernetes.io/projected/c73877a8-533a-4d00-9755-917bee4519d4-kube-api-access-fftwg\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:03.504567 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.504492 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcqjb\" (UniqueName: \"kubernetes.io/projected/246fa19f-ba51-465a-818c-4c19d3786857-kube-api-access-dcqjb\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:03.504567 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.504536 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73877a8-533a-4d00-9755-917bee4519d4-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:03.504567 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:03.504551 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246fa19f-ba51-465a-818c-4c19d3786857-plugins-config-volume\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:04.134881 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.134843 2579 generic.go:356] "Generic (PLEG): container finished" podID="246fa19f-ba51-465a-818c-4c19d3786857" containerID="ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c" exitCode=0 Sep 15 21:16:04.135355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.134896 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" Sep 15 21:16:04.135355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.134936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" event={"ID":"246fa19f-ba51-465a-818c-4c19d3786857","Type":"ContainerDied","Data":"ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c"} Sep 15 21:16:04.135355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.134992 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" event={"ID":"246fa19f-ba51-465a-818c-4c19d3786857","Type":"ContainerDied","Data":"d7c39fbbb007075065d03e524ba6f33d944b548b5e02a038ef54a1ea4147d63b"} Sep 15 21:16:04.135355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.135018 2579 scope.go:117] "RemoveContainer" containerID="ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c" Sep 15 21:16:04.136629 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.136604 2579 generic.go:356] "Generic (PLEG): container finished" podID="c73877a8-533a-4d00-9755-917bee4519d4" containerID="c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0" exitCode=0 Sep 15 21:16:04.136715 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.136654 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" Sep 15 21:16:04.136796 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.136655 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" event={"ID":"c73877a8-533a-4d00-9755-917bee4519d4","Type":"ContainerDied","Data":"c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0"} Sep 15 21:16:04.136859 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.136843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z" event={"ID":"c73877a8-533a-4d00-9755-917bee4519d4","Type":"ContainerDied","Data":"fcd5fc0aa663cf72531d5e2d8f411e9ac3202d8e8413dcd3c7408e65d1c79eab"} Sep 15 21:16:04.145707 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.145686 2579 scope.go:117] "RemoveContainer" containerID="ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c" Sep 15 21:16:04.146046 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:16:04.146028 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c\": container with ID starting with ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c not found: ID does not exist" containerID="ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c" Sep 15 21:16:04.146133 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.146054 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c"} err="failed to get container status \"ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c\": rpc error: code = NotFound desc = could not find container \"ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c\": container with ID starting with ec12123ad3fe4c1a541416838428ec70939769319c01d0959e5328ce9cc9a66c not found: ID does not exist" Sep 15 21:16:04.146133 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.146071 2579 scope.go:117] "RemoveContainer" containerID="c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0" Sep 15 21:16:04.154676 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.154656 2579 scope.go:117] "RemoveContainer" containerID="c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0" Sep 15 21:16:04.154949 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:16:04.154930 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0\": container with ID starting with c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0 not found: ID does not exist" containerID="c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0" Sep 15 21:16:04.155011 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.154961 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0"} err="failed to get container status \"c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0\": rpc error: code = NotFound desc = could not find container \"c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0\": container with ID starting with c595e3493ccbab6877aafe9ffac107cd17bebf44d4b09811e0716b1451cbfaf0 not found: ID does not exist" Sep 15 21:16:04.160773 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.160749 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5"] Sep 15 21:16:04.164449 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.164428 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5"] Sep 15 21:16:04.173707 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.173681 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z"] Sep 15 21:16:04.177315 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.177293 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-280fb4-6657f64bb-xwg6z"] Sep 15 21:16:04.282484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:04.282445 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-280fb4-69695b8c4c-5fhr5" podUID="246fa19f-ba51-465a-818c-4c19d3786857" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.79:9004: i/o timeout" Sep 15 21:16:05.311227 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:05.311194 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="246fa19f-ba51-465a-818c-4c19d3786857" path="/var/lib/kubelet/pods/246fa19f-ba51-465a-818c-4c19d3786857/volumes" Sep 15 21:16:05.311619 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:05.311568 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73877a8-533a-4d00-9755-917bee4519d4" path="/var/lib/kubelet/pods/c73877a8-533a-4d00-9755-917bee4519d4/volumes" Sep 15 21:16:10.218928 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.218884 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm"] Sep 15 21:16:10.219433 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219393 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0250e06-0ed2-4ca7-b98c-73b4dc913dc4" containerName="revoke-keys" Sep 15 21:16:10.219433 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219413 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0250e06-0ed2-4ca7-b98c-73b4dc913dc4" containerName="revoke-keys" Sep 15 21:16:10.219433 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219428 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73877a8-533a-4d00-9755-917bee4519d4" containerName="payload-pre-processing" Sep 15 21:16:10.219654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219438 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73877a8-533a-4d00-9755-917bee4519d4" containerName="payload-pre-processing" Sep 15 21:16:10.219654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219449 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246fa19f-ba51-465a-818c-4c19d3786857" containerName="payload-processing" Sep 15 21:16:10.219654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219457 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="246fa19f-ba51-465a-818c-4c19d3786857" containerName="payload-processing" Sep 15 21:16:10.219654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219515 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" Sep 15 21:16:10.219654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219523 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" Sep 15 21:16:10.219654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219535 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" Sep 15 21:16:10.219654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219543 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" Sep 15 21:16:10.219654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219635 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73877a8-533a-4d00-9755-917bee4519d4" containerName="payload-pre-processing" Sep 15 21:16:10.219654 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219654 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" Sep 15 21:16:10.220072 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219663 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0250e06-0ed2-4ca7-b98c-73b4dc913dc4" containerName="revoke-keys" Sep 15 21:16:10.220072 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219673 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="700944ff-55e6-462f-95b0-1222e07aa47f" containerName="istio-proxy" Sep 15 21:16:10.220072 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.219682 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="246fa19f-ba51-465a-818c-4c19d3786857" containerName="payload-processing" Sep 15 21:16:10.224269 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.224247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" Sep 15 21:16:10.227208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.227179 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:16:10.229081 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.229052 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm"] Sep 15 21:16:10.370361 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.370320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4791079d-49c5-4462-80e9-4c6dddd4fbcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm\" (UID: \"4791079d-49c5-4462-80e9-4c6dddd4fbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" Sep 15 21:16:10.471594 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.471475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4791079d-49c5-4462-80e9-4c6dddd4fbcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm\" (UID: \"4791079d-49c5-4462-80e9-4c6dddd4fbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" Sep 15 21:16:10.472109 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.472089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4791079d-49c5-4462-80e9-4c6dddd4fbcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm\" (UID: \"4791079d-49c5-4462-80e9-4c6dddd4fbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" Sep 15 21:16:10.536290 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.536255 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" Sep 15 21:16:10.806725 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:10.806698 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm"] Sep 15 21:16:10.809101 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:16:10.809072 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4791079d_49c5_4462_80e9_4c6dddd4fbcb.slice/crio-5dd6c746fab6d099638c84f489023c6cae283fd3282337369e843310954bd60a WatchSource:0}: Error finding container 5dd6c746fab6d099638c84f489023c6cae283fd3282337369e843310954bd60a: Status 404 returned error can't find the container with id 5dd6c746fab6d099638c84f489023c6cae283fd3282337369e843310954bd60a Sep 15 21:16:11.179255 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:11.179158 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" event={"ID":"4791079d-49c5-4462-80e9-4c6dddd4fbcb","Type":"ContainerStarted","Data":"179b2a3b08988167756c27d0ee1300a5545018b78c54ff42b99b45f9a2a69988"} Sep 15 21:16:11.179255 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:11.179212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" event={"ID":"4791079d-49c5-4462-80e9-4c6dddd4fbcb","Type":"ContainerStarted","Data":"5dd6c746fab6d099638c84f489023c6cae283fd3282337369e843310954bd60a"} Sep 15 21:16:11.194758 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:11.194699 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" podStartSLOduration=1.194683645 podStartE2EDuration="1.194683645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:11.194014264 +0000 UTC m=+2368.387967719" watchObservedRunningTime="2026-09-15 21:16:11.194683645 +0000 UTC m=+2368.388637099" Sep 15 21:16:12.189041 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:12.189005 2579 generic.go:356] "Generic (PLEG): container finished" podID="4791079d-49c5-4462-80e9-4c6dddd4fbcb" containerID="179b2a3b08988167756c27d0ee1300a5545018b78c54ff42b99b45f9a2a69988" exitCode=0 Sep 15 21:16:12.189474 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:12.189085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" event={"ID":"4791079d-49c5-4462-80e9-4c6dddd4fbcb","Type":"ContainerDied","Data":"179b2a3b08988167756c27d0ee1300a5545018b78c54ff42b99b45f9a2a69988"} Sep 15 21:16:13.325664 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:13.325637 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" Sep 15 21:16:13.500108 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:13.500023 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4791079d-49c5-4462-80e9-4c6dddd4fbcb-maas-api-service-ca\") pod \"4791079d-49c5-4462-80e9-4c6dddd4fbcb\" (UID: \"4791079d-49c5-4462-80e9-4c6dddd4fbcb\") " Sep 15 21:16:13.500468 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:13.500439 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4791079d-49c5-4462-80e9-4c6dddd4fbcb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4791079d-49c5-4462-80e9-4c6dddd4fbcb" (UID: "4791079d-49c5-4462-80e9-4c6dddd4fbcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:16:13.601739 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:13.601698 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4791079d-49c5-4462-80e9-4c6dddd4fbcb-maas-api-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:14.200156 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:14.200115 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" event={"ID":"4791079d-49c5-4462-80e9-4c6dddd4fbcb","Type":"ContainerDied","Data":"5dd6c746fab6d099638c84f489023c6cae283fd3282337369e843310954bd60a"} Sep 15 21:16:14.200156 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:14.200156 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm" Sep 15 21:16:14.200379 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:14.200174 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dd6c746fab6d099638c84f489023c6cae283fd3282337369e843310954bd60a" Sep 15 21:16:21.422696 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:21.422652 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm"] Sep 15 21:16:21.425817 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:21.425789 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-728877-4twpm"] Sep 15 21:16:23.311204 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:23.311171 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4791079d-49c5-4462-80e9-4c6dddd4fbcb" path="/var/lib/kubelet/pods/4791079d-49c5-4462-80e9-4c6dddd4fbcb/volumes" Sep 15 21:16:28.407148 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:28.407115 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl"] Sep 15 21:16:28.407596 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:28.407335 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" containerID="cri-o://70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b" gracePeriod=30 Sep 15 21:16:33.660857 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.660831 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:16:33.780421 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780397 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-socket\") pod \"c52fe772-acba-4dbe-9a12-543c7c3b0955\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " Sep 15 21:16:33.780624 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780440 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52fe772-acba-4dbe-9a12-543c7c3b0955-istiod-ca-cert\") pod \"c52fe772-acba-4dbe-9a12-543c7c3b0955\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " Sep 15 21:16:33.780624 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780466 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-envoy\") pod \"c52fe772-acba-4dbe-9a12-543c7c3b0955\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " Sep 15 21:16:33.780624 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780522 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-podinfo\") pod \"c52fe772-acba-4dbe-9a12-543c7c3b0955\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " Sep 15 21:16:33.780624 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780597 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-data\") pod \"c52fe772-acba-4dbe-9a12-543c7c3b0955\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " Sep 15 21:16:33.780844 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780681 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2xhd\" (UniqueName: \"kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-kube-api-access-k2xhd\") pod \"c52fe772-acba-4dbe-9a12-543c7c3b0955\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " Sep 15 21:16:33.780844 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780747 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-certs\") pod \"c52fe772-acba-4dbe-9a12-543c7c3b0955\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " Sep 15 21:16:33.780844 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780748 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c52fe772-acba-4dbe-9a12-543c7c3b0955" (UID: "c52fe772-acba-4dbe-9a12-543c7c3b0955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:16:33.780844 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780806 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-token\") pod \"c52fe772-acba-4dbe-9a12-543c7c3b0955\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " Sep 15 21:16:33.780844 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780838 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-credential-socket\") pod \"c52fe772-acba-4dbe-9a12-543c7c3b0955\" (UID: \"c52fe772-acba-4dbe-9a12-543c7c3b0955\") " Sep 15 21:16:33.781125 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780970 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c52fe772-acba-4dbe-9a12-543c7c3b0955" (UID: "c52fe772-acba-4dbe-9a12-543c7c3b0955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:16:33.781125 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.780982 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52fe772-acba-4dbe-9a12-543c7c3b0955-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c52fe772-acba-4dbe-9a12-543c7c3b0955" (UID: "c52fe772-acba-4dbe-9a12-543c7c3b0955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:16:33.781125 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.781054 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-data" (OuterVolumeSpecName: "istio-data") pod "c52fe772-acba-4dbe-9a12-543c7c3b0955" (UID: "c52fe772-acba-4dbe-9a12-543c7c3b0955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:16:33.781299 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.781178 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c52fe772-acba-4dbe-9a12-543c7c3b0955" (UID: "c52fe772-acba-4dbe-9a12-543c7c3b0955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:16:33.781546 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.781253 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:33.781546 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.781384 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-workload-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:33.781546 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.781402 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52fe772-acba-4dbe-9a12-543c7c3b0955-istiod-ca-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:33.781546 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.781416 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-data\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:33.782975 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.782943 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c52fe772-acba-4dbe-9a12-543c7c3b0955" (UID: "c52fe772-acba-4dbe-9a12-543c7c3b0955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:16:33.782975 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.782955 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c52fe772-acba-4dbe-9a12-543c7c3b0955" (UID: "c52fe772-acba-4dbe-9a12-543c7c3b0955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:16:33.783167 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.783014 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-kube-api-access-k2xhd" (OuterVolumeSpecName: "kube-api-access-k2xhd") pod "c52fe772-acba-4dbe-9a12-543c7c3b0955" (UID: "c52fe772-acba-4dbe-9a12-543c7c3b0955"). InnerVolumeSpecName "kube-api-access-k2xhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:16:33.783209 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.783189 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-token" (OuterVolumeSpecName: "istio-token") pod "c52fe772-acba-4dbe-9a12-543c7c3b0955" (UID: "c52fe772-acba-4dbe-9a12-543c7c3b0955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:16:33.881889 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.881851 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-envoy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:33.881889 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.881881 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-podinfo\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:33.881889 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.881891 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2xhd\" (UniqueName: \"kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-kube-api-access-k2xhd\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:33.882138 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.881903 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52fe772-acba-4dbe-9a12-543c7c3b0955-istio-token\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:33.882138 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:33.881911 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52fe772-acba-4dbe-9a12-543c7c3b0955-credential-socket\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:16:34.308634 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.308598 2579 generic.go:356] "Generic (PLEG): container finished" podID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerID="70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b" exitCode=0 Sep 15 21:16:34.308850 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.308644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" event={"ID":"c52fe772-acba-4dbe-9a12-543c7c3b0955","Type":"ContainerDied","Data":"70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b"} Sep 15 21:16:34.308850 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.308652 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" Sep 15 21:16:34.308850 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.308693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl" event={"ID":"c52fe772-acba-4dbe-9a12-543c7c3b0955","Type":"ContainerDied","Data":"30baf4d301e76b7ca470bac907b4041aa098f0af0c7cbb0cab4fc44966e3622d"} Sep 15 21:16:34.308850 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.308723 2579 scope.go:117] "RemoveContainer" containerID="70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b" Sep 15 21:16:34.318771 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.318753 2579 scope.go:117] "RemoveContainer" containerID="d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db" Sep 15 21:16:34.326898 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.326875 2579 scope.go:117] "RemoveContainer" containerID="70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b" Sep 15 21:16:34.327174 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:16:34.327153 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b\": container with ID starting with 70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b not found: ID does not exist" containerID="70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b" Sep 15 21:16:34.327221 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.327185 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b"} err="failed to get container status \"70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b\": rpc error: code = NotFound desc = could not find container \"70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b\": container with ID starting with 70c6e9a66310fa3908cc0a8cccc9fd131a42c345b476691d10e91b70c62b373b not found: ID does not exist" Sep 15 21:16:34.327221 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.327203 2579 scope.go:117] "RemoveContainer" containerID="d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db" Sep 15 21:16:34.327508 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:16:34.327460 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db\": container with ID starting with d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db not found: ID does not exist" containerID="d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db" Sep 15 21:16:34.327601 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.327491 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db"} err="failed to get container status \"d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db\": rpc error: code = NotFound desc = could not find container \"d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db\": container with ID starting with d9bb6a8f489f636dd7192bfad941670f20d41deb09130a8fb53d06ab409f03db not found: ID does not exist" Sep 15 21:16:34.330130 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.330107 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl"] Sep 15 21:16:34.333770 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:34.333750 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-728877-openshift-default-5f7f54cbcb-swtnl"] Sep 15 21:16:35.312299 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:16:35.312260 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" path="/var/lib/kubelet/pods/c52fe772-acba-4dbe-9a12-543c7c3b0955/volumes" Sep 15 21:20:43.260299 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.260264 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n"] Sep 15 21:20:43.260756 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.260595 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" Sep 15 21:20:43.260756 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.260607 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" Sep 15 21:20:43.260756 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.260622 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4791079d-49c5-4462-80e9-4c6dddd4fbcb" containerName="revoke-keys" Sep 15 21:20:43.260756 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.260628 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4791079d-49c5-4462-80e9-4c6dddd4fbcb" containerName="revoke-keys" Sep 15 21:20:43.260756 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.260680 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4791079d-49c5-4462-80e9-4c6dddd4fbcb" containerName="revoke-keys" Sep 15 21:20:43.260756 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.260688 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" Sep 15 21:20:43.260756 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.260696 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" Sep 15 21:20:43.263862 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.263838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.266167 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.266144 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9abccb25-kserve-self-signed-certs\"" Sep 15 21:20:43.274537 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.274513 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n"] Sep 15 21:20:43.426158 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.426128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-dshm\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.426367 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.426178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-tmp-dir\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.426367 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.426202 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-home\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.426367 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.426288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pkgf\" (UniqueName: \"kubernetes.io/projected/40322eae-1532-4a10-8f40-023139e7e0a4-kube-api-access-4pkgf\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.426367 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.426345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-model-cache\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.426575 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.426536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40322eae-1532-4a10-8f40-023139e7e0a4-tls-certs\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.527996 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.527964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-tmp-dir\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.527996 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.527999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-home\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.528269 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.528033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pkgf\" (UniqueName: \"kubernetes.io/projected/40322eae-1532-4a10-8f40-023139e7e0a4-kube-api-access-4pkgf\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.528269 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.528073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-model-cache\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.528269 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.528104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40322eae-1532-4a10-8f40-023139e7e0a4-tls-certs\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.528269 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.528125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-dshm\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.528484 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.528423 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-home\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.528578 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.528485 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-tmp-dir\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.528578 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.528542 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-model-cache\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.530622 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.530594 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-dshm\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.530758 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.530740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40322eae-1532-4a10-8f40-023139e7e0a4-tls-certs\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.536010 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.535989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pkgf\" (UniqueName: \"kubernetes.io/projected/40322eae-1532-4a10-8f40-023139e7e0a4-kube-api-access-4pkgf\") pod \"e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.575232 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.575201 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:43.844453 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:43.844397 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n"] Sep 15 21:20:43.846745 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:20:43.846716 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40322eae_1532_4a10_8f40_023139e7e0a4.slice/crio-8041e8afaaa4baffe1bc6d827a3d7337e1e989de9123c6789291839a884c9bee WatchSource:0}: Error finding container 8041e8afaaa4baffe1bc6d827a3d7337e1e989de9123c6789291839a884c9bee: Status 404 returned error can't find the container with id 8041e8afaaa4baffe1bc6d827a3d7337e1e989de9123c6789291839a884c9bee Sep 15 21:20:44.036684 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:44.036654 2579 scope.go:117] "RemoveContainer" containerID="603e23c1da15b13d1bff075e35e15876483d4a9f537cff59541c98ab3a32dc97" Sep 15 21:20:44.045434 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:44.045415 2579 scope.go:117] "RemoveContainer" containerID="d1aeb51d7917c56261162efc5882de556fc09339d2aa718f35d8cc4ee3d2310f" Sep 15 21:20:44.531564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:44.531525 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" event={"ID":"40322eae-1532-4a10-8f40-023139e7e0a4","Type":"ContainerStarted","Data":"835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed"} Sep 15 21:20:44.531979 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:44.531579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" event={"ID":"40322eae-1532-4a10-8f40-023139e7e0a4","Type":"ContainerStarted","Data":"8041e8afaaa4baffe1bc6d827a3d7337e1e989de9123c6789291839a884c9bee"} Sep 15 21:20:44.548306 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:44.548246 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" podStartSLOduration=1.548229262 podStartE2EDuration="1.548229262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:44.546823711 +0000 UTC m=+2641.740777158" watchObservedRunningTime="2026-09-15 21:20:44.548229262 +0000 UTC m=+2641.742182773" Sep 15 21:20:53.575803 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:53.575764 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:53.576300 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:53.575819 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:53.588518 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:53.588471 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:20:54.591750 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:20:54.591720 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:21:06.269380 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.269338 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw"] Sep 15 21:21:06.269849 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.269832 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" Sep 15 21:21:06.269895 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.269851 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52fe772-acba-4dbe-9a12-543c7c3b0955" containerName="istio-proxy" Sep 15 21:21:06.272928 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.272898 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.275254 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.275227 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9abccb25-kserve-self-signed-certs\"" Sep 15 21:21:06.282605 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.282579 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw"] Sep 15 21:21:06.333731 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.333685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tls-certs\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.333731 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.333737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-home\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.334002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.333824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-model-cache\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.334002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.333873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tmp-dir\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.334002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.333920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-dshm\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.334002 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.333946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqpd6\" (UniqueName: \"kubernetes.io/projected/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-kube-api-access-rqpd6\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.434757 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.434708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-home\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.434919 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.434789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-model-cache\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.434919 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.434834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tmp-dir\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.434919 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.434868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-dshm\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.434919 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.434893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqpd6\" (UniqueName: \"kubernetes.io/projected/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-kube-api-access-rqpd6\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.435142 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.434926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tls-certs\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.435202 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.435185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-home\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.435276 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.435229 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-model-cache\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.435355 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.435287 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tmp-dir\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.437327 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.437302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-dshm\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.437510 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.437480 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tls-certs\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.442522 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.442485 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqpd6\" (UniqueName: \"kubernetes.io/projected/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-kube-api-access-rqpd6\") pod \"e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.584096 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.584000 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:06.854933 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:06.854852 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw"] Sep 15 21:21:06.857886 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:21:06.857855 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c48d098_e1e7_42ec_a9ba_078f5a46f52c.slice/crio-62bc42e3ec6f0ad77bbb99942d97f7d2fc424b062de445e7279858480f4a4f3b WatchSource:0}: Error finding container 62bc42e3ec6f0ad77bbb99942d97f7d2fc424b062de445e7279858480f4a4f3b: Status 404 returned error can't find the container with id 62bc42e3ec6f0ad77bbb99942d97f7d2fc424b062de445e7279858480f4a4f3b Sep 15 21:21:07.649520 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:07.649463 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" event={"ID":"2c48d098-e1e7-42ec-a9ba-078f5a46f52c","Type":"ContainerStarted","Data":"ce9de2399c3dad4b22f257ba6b7fcbe192ded267431cd6b310461116c1bb8ffb"} Sep 15 21:21:07.649924 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:07.649542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" event={"ID":"2c48d098-e1e7-42ec-a9ba-078f5a46f52c","Type":"ContainerStarted","Data":"62bc42e3ec6f0ad77bbb99942d97f7d2fc424b062de445e7279858480f4a4f3b"} Sep 15 21:21:07.666612 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:07.666566 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" podStartSLOduration=1.6665474009999999 podStartE2EDuration="1.666547401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:07.665368752 +0000 UTC m=+2664.859322206" watchObservedRunningTime="2026-09-15 21:21:07.666547401 +0000 UTC m=+2664.860500856" Sep 15 21:21:16.584796 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:16.584754 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:16.584796 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:16.584809 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:16.597540 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:16.597492 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:16.706410 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:16.706380 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:29.721990 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:29.721956 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n"] Sep 15 21:21:30.653865 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:30.653830 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw"] Sep 15 21:21:44.068183 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:44.068102 2579 scope.go:117] "RemoveContainer" containerID="78d9eca09c05b51976bd65c872103451fbf1b36f99006a75c3873e0c490e44ef" Sep 15 21:21:44.076559 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:44.076541 2579 scope.go:117] "RemoveContainer" containerID="371ad03319aa21df9e37e6b5406a914949d5602983d60dc5c000a7cab1d1e0a8" Sep 15 21:21:44.742535 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:44.742445 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" podUID="40322eae-1532-4a10-8f40-023139e7e0a4" containerName="main" containerID="cri-o://835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed" gracePeriod=45 Sep 15 21:21:44.994722 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:44.994659 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:21:45.084124 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084076 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40322eae-1532-4a10-8f40-023139e7e0a4-tls-certs\") pod \"40322eae-1532-4a10-8f40-023139e7e0a4\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " Sep 15 21:21:45.084638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084163 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-home\") pod \"40322eae-1532-4a10-8f40-023139e7e0a4\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " Sep 15 21:21:45.084638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084358 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pkgf\" (UniqueName: \"kubernetes.io/projected/40322eae-1532-4a10-8f40-023139e7e0a4-kube-api-access-4pkgf\") pod \"40322eae-1532-4a10-8f40-023139e7e0a4\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " Sep 15 21:21:45.084638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084362 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-home" (OuterVolumeSpecName: "home") pod "40322eae-1532-4a10-8f40-023139e7e0a4" (UID: "40322eae-1532-4a10-8f40-023139e7e0a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:45.084638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084431 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-model-cache\") pod \"40322eae-1532-4a10-8f40-023139e7e0a4\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " Sep 15 21:21:45.084638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084467 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-tmp-dir\") pod \"40322eae-1532-4a10-8f40-023139e7e0a4\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " Sep 15 21:21:45.084638 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084521 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-dshm\") pod \"40322eae-1532-4a10-8f40-023139e7e0a4\" (UID: \"40322eae-1532-4a10-8f40-023139e7e0a4\") " Sep 15 21:21:45.084900 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084662 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "40322eae-1532-4a10-8f40-023139e7e0a4" (UID: "40322eae-1532-4a10-8f40-023139e7e0a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:45.084900 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084725 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40322eae-1532-4a10-8f40-023139e7e0a4" (UID: "40322eae-1532-4a10-8f40-023139e7e0a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:45.084900 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084874 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:45.085007 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084908 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:45.085007 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.084918 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:45.086612 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.086591 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40322eae-1532-4a10-8f40-023139e7e0a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40322eae-1532-4a10-8f40-023139e7e0a4" (UID: "40322eae-1532-4a10-8f40-023139e7e0a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:21:45.086709 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.086687 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40322eae-1532-4a10-8f40-023139e7e0a4-kube-api-access-4pkgf" (OuterVolumeSpecName: "kube-api-access-4pkgf") pod "40322eae-1532-4a10-8f40-023139e7e0a4" (UID: "40322eae-1532-4a10-8f40-023139e7e0a4"). InnerVolumeSpecName "kube-api-access-4pkgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:21:45.086749 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.086687 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-dshm" (OuterVolumeSpecName: "dshm") pod "40322eae-1532-4a10-8f40-023139e7e0a4" (UID: "40322eae-1532-4a10-8f40-023139e7e0a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:45.185523 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.185462 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40322eae-1532-4a10-8f40-023139e7e0a4-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:45.185701 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.185520 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pkgf\" (UniqueName: \"kubernetes.io/projected/40322eae-1532-4a10-8f40-023139e7e0a4-kube-api-access-4pkgf\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:45.185701 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.185547 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40322eae-1532-4a10-8f40-023139e7e0a4-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:45.674012 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.673902 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" podUID="2c48d098-e1e7-42ec-a9ba-078f5a46f52c" containerName="main" containerID="cri-o://ce9de2399c3dad4b22f257ba6b7fcbe192ded267431cd6b310461116c1bb8ffb" gracePeriod=45 Sep 15 21:21:45.841143 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.841111 2579 generic.go:356] "Generic (PLEG): container finished" podID="40322eae-1532-4a10-8f40-023139e7e0a4" containerID="835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed" exitCode=0 Sep 15 21:21:45.841341 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.841155 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" Sep 15 21:21:45.841341 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.841188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" event={"ID":"40322eae-1532-4a10-8f40-023139e7e0a4","Type":"ContainerDied","Data":"835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed"} Sep 15 21:21:45.841341 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.841236 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n" event={"ID":"40322eae-1532-4a10-8f40-023139e7e0a4","Type":"ContainerDied","Data":"8041e8afaaa4baffe1bc6d827a3d7337e1e989de9123c6789291839a884c9bee"} Sep 15 21:21:45.841341 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.841267 2579 scope.go:117] "RemoveContainer" containerID="835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed" Sep 15 21:21:45.848583 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.848550 2579 generic.go:356] "Generic (PLEG): container finished" podID="2c48d098-e1e7-42ec-a9ba-078f5a46f52c" containerID="ce9de2399c3dad4b22f257ba6b7fcbe192ded267431cd6b310461116c1bb8ffb" exitCode=0 Sep 15 21:21:45.848703 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.848608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" event={"ID":"2c48d098-e1e7-42ec-a9ba-078f5a46f52c","Type":"ContainerDied","Data":"ce9de2399c3dad4b22f257ba6b7fcbe192ded267431cd6b310461116c1bb8ffb"} Sep 15 21:21:45.856005 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.855983 2579 scope.go:117] "RemoveContainer" containerID="835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed" Sep 15 21:21:45.856396 ip-10-0-130-107 kubenswrapper[2579]: E0915 21:21:45.856362 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed\": container with ID starting with 835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed not found: ID does not exist" containerID="835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed" Sep 15 21:21:45.856544 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.856407 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed"} err="failed to get container status \"835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed\": rpc error: code = NotFound desc = could not find container \"835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed\": container with ID starting with 835ff2d2968502a8131f890e992e59963146b9e71d496ed7c21618272dc0deed not found: ID does not exist" Sep 15 21:21:45.867108 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.867080 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n"] Sep 15 21:21:45.868930 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.868907 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9abccb25-kserve-7d8db797dc-zsj4n"] Sep 15 21:21:45.930113 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.930048 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:45.994059 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994024 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-home\") pod \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " Sep 15 21:21:45.994059 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994063 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-model-cache\") pod \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " Sep 15 21:21:45.994320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994148 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tmp-dir\") pod \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " Sep 15 21:21:45.994320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994227 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-dshm\") pod \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " Sep 15 21:21:45.994320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994256 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqpd6\" (UniqueName: \"kubernetes.io/projected/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-kube-api-access-rqpd6\") pod \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " Sep 15 21:21:45.994320 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994290 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tls-certs\") pod \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\" (UID: \"2c48d098-e1e7-42ec-a9ba-078f5a46f52c\") " Sep 15 21:21:45.994564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-home" (OuterVolumeSpecName: "home") pod "2c48d098-e1e7-42ec-a9ba-078f5a46f52c" (UID: "2c48d098-e1e7-42ec-a9ba-078f5a46f52c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:45.994564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994339 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c48d098-e1e7-42ec-a9ba-078f5a46f52c" (UID: "2c48d098-e1e7-42ec-a9ba-078f5a46f52c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:45.994564 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994422 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c48d098-e1e7-42ec-a9ba-078f5a46f52c" (UID: "2c48d098-e1e7-42ec-a9ba-078f5a46f52c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:45.994700 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994582 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-home\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:45.994700 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994596 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:45.994700 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.994609 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tmp-dir\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:45.996481 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.996452 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-dshm" (OuterVolumeSpecName: "dshm") pod "2c48d098-e1e7-42ec-a9ba-078f5a46f52c" (UID: "2c48d098-e1e7-42ec-a9ba-078f5a46f52c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:45.996621 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.996485 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c48d098-e1e7-42ec-a9ba-078f5a46f52c" (UID: "2c48d098-e1e7-42ec-a9ba-078f5a46f52c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:21:45.996621 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:45.996534 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-kube-api-access-rqpd6" (OuterVolumeSpecName: "kube-api-access-rqpd6") pod "2c48d098-e1e7-42ec-a9ba-078f5a46f52c" (UID: "2c48d098-e1e7-42ec-a9ba-078f5a46f52c"). InnerVolumeSpecName "kube-api-access-rqpd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:21:46.095709 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:46.095670 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-dshm\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:46.095709 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:46.095705 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqpd6\" (UniqueName: \"kubernetes.io/projected/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-kube-api-access-rqpd6\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:46.095709 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:46.095715 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c48d098-e1e7-42ec-a9ba-078f5a46f52c-tls-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\"" Sep 15 21:21:46.856416 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:46.856388 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" Sep 15 21:21:46.856416 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:46.856410 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw" event={"ID":"2c48d098-e1e7-42ec-a9ba-078f5a46f52c","Type":"ContainerDied","Data":"62bc42e3ec6f0ad77bbb99942d97f7d2fc424b062de445e7279858480f4a4f3b"} Sep 15 21:21:46.856690 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:46.856463 2579 scope.go:117] "RemoveContainer" containerID="ce9de2399c3dad4b22f257ba6b7fcbe192ded267431cd6b310461116c1bb8ffb" Sep 15 21:21:46.879919 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:46.879894 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw"] Sep 15 21:21:46.883194 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:46.883166 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9abccb25-kserve-84c69b4447-bc8mw"] Sep 15 21:21:47.311749 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:47.311718 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c48d098-e1e7-42ec-a9ba-078f5a46f52c" path="/var/lib/kubelet/pods/2c48d098-e1e7-42ec-a9ba-078f5a46f52c/volumes" Sep 15 21:21:47.312140 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:21:47.312101 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40322eae-1532-4a10-8f40-023139e7e0a4" path="/var/lib/kubelet/pods/40322eae-1532-4a10-8f40-023139e7e0a4/volumes" Sep 15 21:22:21.818250 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:21.818209 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74fdb965b6-f4b85_2f4f92f2-3282-49e0-8f0b-ae7928c240b9/authorino/0.log" Sep 15 21:22:28.352208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:28.352175 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-tm88q_42a75d67-bde1-4726-839b-cb9faa24cb0e/manager/0.log" Sep 15 21:22:28.359157 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:28.359136 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-tm88q_42a75d67-bde1-4726-839b-cb9faa24cb0e/copy-manifests/0.log" Sep 15 21:22:28.970289 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:28.970262 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825100-84zz7_41af4c96-d2fa-45d4-9157-4ce7a7e83014/cleanup/0.log" Sep 15 21:22:29.086349 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:29.086316 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825115-2j9p2_12d4f7a1-61ec-4a22-822a-77830205301e/cleanup/0.log" Sep 15 21:22:29.209860 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:29.209825 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-xhn8s_eaf70320-ef57-4195-9f6f-fb7da3b9575a/postgres/0.log" Sep 15 21:22:30.039012 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:30.038980 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74fdb965b6-f4b85_2f4f92f2-3282-49e0-8f0b-ae7928c240b9/authorino/0.log" Sep 15 21:22:30.791606 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:30.791575 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-zc59d_9fb4446e-1739-4825-bfb5-04c425e6cb44/manager/0.log" Sep 15 21:22:31.542013 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:31.541978 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65544b76bd-m4gxx_08c512b4-9c74-4fec-8cac-427b40788b97/kube-auth-proxy/0.log" Sep 15 21:22:32.393789 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:32.393706 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-f7bdn_1396b69c-8226-49b6-a226-e71568470df2/main/0.log" Sep 15 21:22:32.993680 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:32.993646 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-n8hng_f53f7be1-d8a4-43fd-8d12-95396993655b/main/0.log" Sep 15 21:22:33.108908 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:33.108873 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-rsvrb_04987fbe-82b1-4e39-b32c-2144082fb22d/main/0.log" Sep 15 21:22:41.864377 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:41.864342 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h48b4_609fe877-8116-4e94-a791-ab35d08392e1/global-pull-secret-syncer/0.log" Sep 15 21:22:41.990446 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:41.990419 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9mm89_d4e3d5cf-fff4-45f3-b820-70c6baee1155/konnectivity-agent/0.log" Sep 15 21:22:42.041022 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:42.040993 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-107.ec2.internal_7d376dec3d60bed61757396e0acedf32/haproxy/0.log" Sep 15 21:22:44.098702 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:44.098670 2579 scope.go:117] "RemoveContainer" containerID="179b2a3b08988167756c27d0ee1300a5545018b78c54ff42b99b45f9a2a69988" Sep 15 21:22:44.107390 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:44.107368 2579 scope.go:117] "RemoveContainer" containerID="e1d3dcd37c4c03b1abe0540d348a97bf879b5e77a9857d7adbbcbae509149389" Sep 15 21:22:46.842350 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:46.842307 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74fdb965b6-f4b85_2f4f92f2-3282-49e0-8f0b-ae7928c240b9/authorino/0.log" Sep 15 21:22:47.057015 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:47.056977 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-zc59d_9fb4446e-1739-4825-bfb5-04c425e6cb44/manager/0.log" Sep 15 21:22:48.920543 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:48.920420 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-xr58z_f7a7aacf-c2e6-40ba-bd51-c943817eda80/cluster-monitoring-operator/0.log" Sep 15 21:22:49.023712 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.023683 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d9d59fdf9-4kj7g_0c6e358f-b24a-4510-9a92-fb290819e97a/metrics-server/0.log" Sep 15 21:22:49.051835 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.051796 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-cmn6m_ba8fe8ea-5355-4a6a-8551-140f398a10a3/monitoring-plugin/0.log" Sep 15 21:22:49.244076 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.243997 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jsgz8_1266b01d-c256-4e1f-b4b1-c613c4168b83/node-exporter/0.log" Sep 15 21:22:49.263473 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.263425 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jsgz8_1266b01d-c256-4e1f-b4b1-c613c4168b83/kube-rbac-proxy/0.log" Sep 15 21:22:49.283818 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.283789 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jsgz8_1266b01d-c256-4e1f-b4b1-c613c4168b83/init-textfile/0.log" Sep 15 21:22:49.425208 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.425177 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6470ac0a-6161-4495-bb61-5f67f4e2de14/prometheus/0.log" Sep 15 21:22:49.445610 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.445585 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6470ac0a-6161-4495-bb61-5f67f4e2de14/config-reloader/0.log" Sep 15 21:22:49.466637 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.466601 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6470ac0a-6161-4495-bb61-5f67f4e2de14/thanos-sidecar/0.log" Sep 15 21:22:49.487358 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.487332 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6470ac0a-6161-4495-bb61-5f67f4e2de14/kube-rbac-proxy-web/0.log" Sep 15 21:22:49.507277 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.507186 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6470ac0a-6161-4495-bb61-5f67f4e2de14/kube-rbac-proxy/0.log" Sep 15 21:22:49.529591 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.529558 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6470ac0a-6161-4495-bb61-5f67f4e2de14/kube-rbac-proxy-thanos/0.log" Sep 15 21:22:49.550538 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.550486 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6470ac0a-6161-4495-bb61-5f67f4e2de14/init-config-reloader/0.log" Sep 15 21:22:49.624557 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:49.624530 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-wzjnq_649b7cd2-d289-4a44-928a-1bb1968a17fc/prometheus-operator-admission-webhook/0.log" Sep 15 21:22:51.105976 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.105937 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-vg485_b799debb-e4ae-47d9-9376-66b29a395c40/networking-console-plugin/0.log" Sep 15 21:22:51.748134 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.748098 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4"] Sep 15 21:22:51.748468 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.748456 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40322eae-1532-4a10-8f40-023139e7e0a4" containerName="main" Sep 15 21:22:51.748548 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.748473 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40322eae-1532-4a10-8f40-023139e7e0a4" containerName="main" Sep 15 21:22:51.748548 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.748492 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c48d098-e1e7-42ec-a9ba-078f5a46f52c" containerName="main" Sep 15 21:22:51.748548 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.748514 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c48d098-e1e7-42ec-a9ba-078f5a46f52c" containerName="main" Sep 15 21:22:51.748647 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.748587 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="40322eae-1532-4a10-8f40-023139e7e0a4" containerName="main" Sep 15 21:22:51.748647 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.748597 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c48d098-e1e7-42ec-a9ba-078f5a46f52c" containerName="main" Sep 15 21:22:51.751711 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.751689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.753711 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.753687 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcx6s\"/\"kube-root-ca.crt\"" Sep 15 21:22:51.754554 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.754533 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcx6s\"/\"openshift-service-ca.crt\"" Sep 15 21:22:51.754636 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.754580 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tcx6s\"/\"default-dockercfg-8qfvh\"" Sep 15 21:22:51.760791 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.760770 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4"] Sep 15 21:22:51.891071 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.891036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-sys\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.891308 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.891090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdx49\" (UniqueName: \"kubernetes.io/projected/d7553d9d-51b8-4934-94a6-705e9b3ccb74-kube-api-access-sdx49\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.891308 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.891160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-podres\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.891308 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.891190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-lib-modules\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.891308 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.891259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-proc\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.992577 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.992522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-sys\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.992778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.992603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdx49\" (UniqueName: \"kubernetes.io/projected/d7553d9d-51b8-4934-94a6-705e9b3ccb74-kube-api-access-sdx49\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.992778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.992664 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-sys\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.992778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.992687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-podres\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.992778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.992730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-lib-modules\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.992778 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.992749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-proc\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.993012 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.992833 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-proc\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.993012 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.992867 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-lib-modules\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:51.993012 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:51.992870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7553d9d-51b8-4934-94a6-705e9b3ccb74-podres\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:52.000567 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:52.000484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdx49\" (UniqueName: \"kubernetes.io/projected/d7553d9d-51b8-4934-94a6-705e9b3ccb74-kube-api-access-sdx49\") pod \"perf-node-gather-daemonset-k9zf4\" (UID: \"d7553d9d-51b8-4934-94a6-705e9b3ccb74\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:52.063376 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:52.063342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:52.332723 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:52.332696 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4"] Sep 15 21:22:52.335358 ip-10-0-130-107 kubenswrapper[2579]: W0915 21:22:52.335321 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd7553d9d_51b8_4934_94a6_705e9b3ccb74.slice/crio-dc6617a82655f89c8c80407699b1993147da5d125d96414cc51fdf8a41d0f571 WatchSource:0}: Error finding container dc6617a82655f89c8c80407699b1993147da5d125d96414cc51fdf8a41d0f571: Status 404 returned error can't find the container with id dc6617a82655f89c8c80407699b1993147da5d125d96414cc51fdf8a41d0f571 Sep 15 21:22:53.183353 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:53.183316 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" event={"ID":"d7553d9d-51b8-4934-94a6-705e9b3ccb74","Type":"ContainerStarted","Data":"5cabc568608e01add9bb940f3cf4f82b2442922bafbf2d8fe48513b12edae6c4"} Sep 15 21:22:53.183586 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:53.183408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" event={"ID":"d7553d9d-51b8-4934-94a6-705e9b3ccb74","Type":"ContainerStarted","Data":"dc6617a82655f89c8c80407699b1993147da5d125d96414cc51fdf8a41d0f571"} Sep 15 21:22:53.183586 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:53.183529 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:22:53.199711 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:53.199660 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" podStartSLOduration=2.1996434049999998 podStartE2EDuration="2.199643405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:53.198800232 +0000 UTC m=+2770.392753688" watchObservedRunningTime="2026-09-15 21:22:53.199643405 +0000 UTC m=+2770.393596860" Sep 15 21:22:54.169999 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:54.169970 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vpbz5_4bb580a9-dcac-4ef7-b922-f5d33cb41155/dns/0.log" Sep 15 21:22:54.190393 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:54.190367 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vpbz5_4bb580a9-dcac-4ef7-b922-f5d33cb41155/kube-rbac-proxy/0.log" Sep 15 21:22:54.234009 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:54.233978 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6t5mb_fa34b802-6f15-48e4-bf1b-fb01f9c2feb3/dns-node-resolver/0.log" Sep 15 21:22:54.787698 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:54.787671 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-895fv_c22bdf6e-dac6-4791-b78d-f713a85241ea/node-ca/0.log" Sep 15 21:22:56.222952 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:56.222924 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65544b76bd-m4gxx_08c512b4-9c74-4fec-8cac-427b40788b97/kube-auth-proxy/0.log" Sep 15 21:22:57.286552 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:57.286523 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wwhnn_ccf27f42-5fe5-4492-a1b7-dfd1d26959a0/serve-healthcheck-canary/0.log" Sep 15 21:22:57.796796 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:57.796767 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4246q_a3e7a137-86c0-4bbf-a96e-718e87443852/kube-rbac-proxy/0.log" Sep 15 21:22:57.816082 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:57.816050 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4246q_a3e7a137-86c0-4bbf-a96e-718e87443852/exporter/0.log" Sep 15 21:22:57.834491 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:57.834460 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4246q_a3e7a137-86c0-4bbf-a96e-718e87443852/extractor/0.log" Sep 15 21:22:59.198679 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:22:59.198645 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-k9zf4" Sep 15 21:23:00.838285 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:00.838252 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-tm88q_42a75d67-bde1-4726-839b-cb9faa24cb0e/manager/0.log" Sep 15 21:23:00.855979 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:00.855957 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-tm88q_42a75d67-bde1-4726-839b-cb9faa24cb0e/copy-manifests/0.log" Sep 15 21:23:02.242618 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:02.242585 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-675cf9bd68-w7zjj_887dee56-230a-46cc-83c7-c943667b4799/manager/0.log" Sep 15 21:23:02.406365 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:02.406320 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65544b76bd-m4gxx_08c512b4-9c74-4fec-8cac-427b40788b97/kube-auth-proxy/0.log" Sep 15 21:23:04.465898 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:04.465817 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65544b76bd-m4gxx_08c512b4-9c74-4fec-8cac-427b40788b97/kube-auth-proxy/0.log" Sep 15 21:23:07.320784 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:07.320754 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-kzx64_adb5bd85-37c1-4626-a581-e952a9409205/migrator/0.log" Sep 15 21:23:07.338905 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:07.338885 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-kzx64_adb5bd85-37c1-4626-a581-e952a9409205/graceful-termination/0.log" Sep 15 21:23:07.688422 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:07.688343 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-wfzch_167ff948-f63b-42d8-b67e-882ddafb2a0d/kube-storage-version-migrator-operator/1.log" Sep 15 21:23:07.690078 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:07.690051 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-wfzch_167ff948-f63b-42d8-b67e-882ddafb2a0d/kube-storage-version-migrator-operator/0.log" Sep 15 21:23:08.646728 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:08.646691 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7qblf_e29a808c-8900-425b-b41b-902affb6253c/kube-multus/0.log" Sep 15 21:23:08.682037 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:08.682007 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n5z2v_0309b1b6-8ef8-4242-a7b0-dcd5bccb9694/kube-multus-additional-cni-plugins/0.log" Sep 15 21:23:08.715536 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:08.715483 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n5z2v_0309b1b6-8ef8-4242-a7b0-dcd5bccb9694/egress-router-binary-copy/0.log" Sep 15 21:23:08.745120 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:08.745095 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n5z2v_0309b1b6-8ef8-4242-a7b0-dcd5bccb9694/cni-plugins/0.log" Sep 15 21:23:08.780948 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:08.780919 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n5z2v_0309b1b6-8ef8-4242-a7b0-dcd5bccb9694/bond-cni-plugin/0.log" Sep 15 21:23:08.817016 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:08.816993 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n5z2v_0309b1b6-8ef8-4242-a7b0-dcd5bccb9694/routeoverride-cni/0.log" Sep 15 21:23:08.851979 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:08.851951 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n5z2v_0309b1b6-8ef8-4242-a7b0-dcd5bccb9694/whereabouts-cni-bincopy/0.log" Sep 15 21:23:08.884948 ip-10-0-130-107 kubenswrapper[2579]: I0915 21:23:08.884921 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n5z2v_0309b1b6-8ef8-4242-a7b0-dcd5bccb9694/whereabouts-cni/0.log"