Sep 17 18:27:06.981216 ip-10-0-131-128 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 18:27:06.981228 ip-10-0-131-128 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 18:27:06.981237 ip-10-0-131-128 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 18:27:06.981425 ip-10-0-131-128 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 18:27:17.101408 ip-10-0-131-128 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 18:27:17.101425 ip-10-0-131-128 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ec83a9b27b1d49d1838ad7613a073a25 --
Sep 17 18:29:23.350196 ip-10-0-131-128 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:29:27.013519 ip-10-0-131-128 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:27.013519 ip-10-0-131-128 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:29:27.013519 ip-10-0-131-128 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:27.013519 ip-10-0-131-128 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:29:27.013519 ip-10-0-131-128 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:27.014405 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.013592    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:29:27.018505 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018481    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:27.018505 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018498    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:27.018505 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018503    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:27.018505 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018507    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:27.018505 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018510    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:27.018505 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018513    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018518    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018522    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018525    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018528    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018532    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018534    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018537    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018540    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018543    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018545    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018548    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018551    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018554    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018557    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018560    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018562    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018565    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018567    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018570    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:27.018726 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018578    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018581    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018583    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018586    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018588    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018591    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018595    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018597    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018600    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018603    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018605    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018608    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018611    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018613    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018616    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018619    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018622    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018625    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018630    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:27.019229 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018633    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018636    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018639    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018642    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018644    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018647    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018650    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018655    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018659    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018662    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018665    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018667    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018670    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018672    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018675    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018678    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018680    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018683    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018685    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018688    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:27.019724 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018690    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018693    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018695    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018698    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018701    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018703    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018706    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018710    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018713    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018716    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018719    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018721    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018724    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018727    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018729    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018732    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018735    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018738    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018740    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:27.020219 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018744    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018746    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018749    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018751    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018754    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018756    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018759    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018762    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018764    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018766    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018769    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018771    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018774    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018776    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018779    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.018781    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019478    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019486    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019489    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019492    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:27.020678 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019495    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019498    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019501    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019503    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019506    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019509    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019512    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019514    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019517    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019520    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019523    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019526    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019528    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019531    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019534    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019537    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019539    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019542    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019545    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:27.021190 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019547    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019550    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019552    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019555    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019557    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019560    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019562    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019565    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019567    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019570    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019572    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019575    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019578    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019581    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019584    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019587    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019589    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019592    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019594    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019597    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:27.021665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019599    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019602    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019604    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019607    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019609    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019612    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019615    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019619    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019621    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019624    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019627    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019629    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019632    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019634    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019637    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019640    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019642    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019645    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019648    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019651    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:27.022172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019653    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019656    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019659    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019662    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019664    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019667    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019670    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019673    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019675    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019678    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019680    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019683    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019686    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019688    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019691    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019694    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019697    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019699    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019702    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019704    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:27.022660 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019708    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019710    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019713    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019717    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019721    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019724    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019728    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019731    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019734    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019737    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019739    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019744    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019747    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019751    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019755    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.019758    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020140    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020145    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020148    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:27.023173 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020151    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020154    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020157    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020159    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020162    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020165    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020169    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020171    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020174    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020177    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020179    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020182    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020184    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020192    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020195    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020198    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020201    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020203    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020206    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020208    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:27.023650 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020211    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020214    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020216    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020219    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020224    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020227    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020230    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020232    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020235    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020238    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020241    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020244    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020247    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020250    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020252    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020255    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020258    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020261    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020264    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020267    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:27.024177 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020270    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020272    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020275    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020278    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020280    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020283    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020286    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020289    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020291    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020294    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020296    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020298    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020301    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020304    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020306    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020309    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020312    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020315    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020317    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020320    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:27.024699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020322    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020325    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020329    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020332    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020334    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020337    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020339    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020342    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020344    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020348    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020352    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020355    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020358    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020361    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020363    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020366    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020370    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020372    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:27.025205 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020375    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020378    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020381    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020383    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020386    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020392    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020395    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020398    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020401    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020404    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020407    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020410    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020413    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020415    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020418    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020420    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020424    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.020427    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020500    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020540    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:29:27.025641 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020546    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020551    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020555    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020558    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020562    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020567    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020570    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020573    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020576    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020580    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020583    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020586    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020588    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020591    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020594    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020597    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020601    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020604    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020607    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020610    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020613    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020617    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020620    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:29:27.026127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020623    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020627    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020630    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020633    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020636    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020640    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020643    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020647    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020650    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020653    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020657    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020660    2577 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020663    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020668    2577 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020671    2577 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020674    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020677    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020680    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020684    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020687    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020690    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020693    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020696    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020699    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020702    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:29:27.026713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020705    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020708    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020711    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020714    2577 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020717    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020721    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020724    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020727    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020730    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020733    2577 flags.go:64] FLAG: --help="false"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020735    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020738    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020742    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020745    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020748    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020751    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020754    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020757    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020760    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020762    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020765    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020768    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020771    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020774    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:29:27.027355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020777    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020780    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020782    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020785    2577 flags.go:64] FLAG: --lock-file=""
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020788    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020790    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020793    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020798    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020802    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020805    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020808    2577 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020810    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020814    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020816    2577 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020821    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020825    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020828    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020832    2577 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020835    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020838    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020841    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020844    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020847    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020850    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020872    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:29:27.027984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020879    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020882    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020885    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020888    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020891    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020898    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020913    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020917    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020920    2577 flags.go:64] FLAG: --port="10250"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020923    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020943    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b02b934b6bdb57ac"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020947    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020951    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020954    2577 flags.go:64] FLAG: --register-node="true"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020957    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020961    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020970    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020973    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020976    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020979    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020983    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020986    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020990    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020992    2577 flags.go:64] FLAG: --runonce="false"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020995    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:29:27.028596 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.020998    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021001    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021004    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021007    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021010    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021014    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021017    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021019    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021022    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021025    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021028    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021031    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021033    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021036    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021042    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021044    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021047    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021051    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021054    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021057    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021060    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021063    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021066    2577 flags.go:64] FLAG: --v="2"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021070    2577 flags.go:64] FLAG: --version="false"
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021074    2577 flags.go:64] FLAG: --vmodule=""
Sep 17 18:29:27.029231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021078    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021081    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021191    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021195    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021200    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021203    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021207    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021210    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021213    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021216    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021219    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021222    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021224    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021227    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021230    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021232    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021235    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021237    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021240    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021243    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:27.029839 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021246    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021248    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021251    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021254    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021256    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021259    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021261    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021264    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021266    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021269    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021272    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021276    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021283    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021286    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021289    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021292    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021295    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021298    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021301    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:27.030349 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021303    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021306    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021310    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021313    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021315    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021318    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021320    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021323    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021326    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021328    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021331    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021334    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021336    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021339    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021341    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021344    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021347    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021360    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021362    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021365    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:27.030821 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021368    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021370    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021373    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021375    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021378    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021381    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021384    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021387    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021389    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021393    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021396    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021399    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021401    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021404    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021408    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021411    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021413    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021416    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021419    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021421    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:27.031344 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021424    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021427    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021429    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021433    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021436    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021440    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021444    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021447    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021450    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021453    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021457    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021460    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021462    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021466    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021469    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021473    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021476    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021479    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021482    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:27.031836 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021488    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021491    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.021496    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.021501    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.028881    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.028896    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.028928    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.028933    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.028992    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.028998    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029001    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029005    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029008    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029011    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029014    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:27.032314 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029017    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029019    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029022    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029025    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029029    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029033    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029036    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029039    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029041    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029044    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029047    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029049    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029052    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029056    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029060    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029063    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029066    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029070    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029073    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029075    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:27.032693 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029078    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029082    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029085    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029088    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029091    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029094    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029096    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029099    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029102    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029104    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029107    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029110    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029113    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029116    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029119    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029121    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029124    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029126    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029129    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:27.033194 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029132    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029134    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029137    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029139    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029142    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029144    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029147    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029150    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029152    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029155    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029158    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029161    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029163    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029166    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029171    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029174    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029178    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029181    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029184    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029186    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:27.033652 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029189    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029192    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029195    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029198    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029200    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029203    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029206    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029208    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029211    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029213    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029216    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029218    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029221    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029224    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029226    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029229    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029231    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029234    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029237    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029240    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:27.034156 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029243    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029246    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029249    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029251    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029254    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029257    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029260    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029263    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029266    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029268    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029271    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029273    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029276    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.029281    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029404    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:27.034644 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029409    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029412    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029415    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029418    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029421    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029424    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029428    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029432    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029435    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029438    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029441    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029444    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029447    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029450    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029453    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029456    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029459    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029462    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029465    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:27.035020 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029467    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029470    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029473    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029477    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029479    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029483    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029485    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029488    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029491    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029494    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029496    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029499    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029502    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029504    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029507    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029511    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029514    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029517    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029520    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029523    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:27.035476 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029525    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029528    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029531    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029533    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029536    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029539    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029541    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029544    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029546    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029549    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029552    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029554    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029557    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029560    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029563    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029565    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029568    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029572    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029575    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029577    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:27.036181 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029580    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029582    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029585    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029587    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029590    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029593    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029595    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029598    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029601    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029603    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029606    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029608    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029611    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029614    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029616    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029620    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029623    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029626    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029629    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029631    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:27.036715 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029634    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029637    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029640    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029642    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029645    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029647    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029650    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029652    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029655    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029658    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029661    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029664    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029666    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029669    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029671    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029674    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029676    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029679    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:27.037342 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:27.029682    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:27.038043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.029687    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:27.038043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.029777    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:29:27.038043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.031761    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:29:27.038043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.032805    2577 server.go:1019] "Starting client certificate rotation"
Sep 17 18:29:27.038043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.032913    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:27.038043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.032955    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:27.038043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.035482    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:27.038043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.036604    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:27.051610 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.051584    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:29:27.054140 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.054123    2577 log.go:25] "Validated CRI v1 image API"
Sep 17 18:29:27.056205 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.056190    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:29:27.058876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.058855    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 864b4b74-88d4-4273-87ce-70d1d54ef13d:/dev/nvme0n1p4 e87da519-0ae5-4e28-9a5c-481b9b99060a:/dev/nvme0n1p3]
Sep 17 18:29:27.058972 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.058874    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:29:27.063059 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.062954    2577 manager.go:216] Machine: {Timestamp:2026-09-17 18:29:27.06248166 +0000 UTC m=+2.108884574 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100941 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec233b958dca0686b5661d7ada4cf5a9 SystemUUID:ec233b95-8dca-0686-b566-1d7ada4cf5a9 BootID:ec83a9b2-7b1d-49d1-838a-d7613a073a25 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:43:1f:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:43:1f:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:89:36:54:18:36 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:29:27.063059 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.063050    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:29:27.063229 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.063119    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:29:27.065105 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.065076    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:29:27.065233 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.065108    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:29:27.066289 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.066275    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:29:27.066289 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.066291    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:29:27.066402 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.066299    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:27.066402 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.066312    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:27.066402 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.066376    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:29:27.069755 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.069742    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:27.069870 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.069861    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:29:27.069928 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.069891    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:29:27.069928 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.069926    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:29:27.070007 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.069949    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:29:27.070007 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.069964    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:29:27.070007 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.069986    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:29:27.070354 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.070342    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:27.070394 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.070361    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:27.071449 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.071431    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:27.072576 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.072559    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:29:27.077026 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.077007    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:29:27.077123 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.077038    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:29:27.078572 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078546    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:29:27.078572 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078570    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:29:27.078669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078578    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:29:27.078669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078586    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:29:27.078669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078595    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:29:27.078669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078604    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:29:27.078669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078623    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:29:27.078669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078629    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:29:27.078669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078636    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:29:27.078669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078641    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:29:27.078669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078650    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:29:27.078669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078656    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:29:27.078978 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078684    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:29:27.078978 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.078697    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:29:27.083523 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.083503    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:29:27.083523 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.083521    2577 server.go:1268] "Started kubelet"
Sep 17 18:29:27.083666 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.083609    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:29:27.083795 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.083603    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:29:27.083844 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.083819    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:29:27.084311 ip-10-0-131-128 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:29:27.084703 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.084360    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:29:27.085336 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.085154    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:29:27.086406 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.086382    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:29:27.086690 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.086676    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:29:27.088695 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.088676    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:29:27.091924 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.091889    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:29:27.091998 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.091919    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:27.092060 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.091998    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:27.092529 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.092502    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:29:27.092606 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.092546    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:29:27.092606 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.092559    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:29:27.092729 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.092711    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:29:27.092729 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.092725    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:29:27.092857 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.092789    2577 factory.go:55] Registering systemd factory
Sep 17 18:29:27.092857 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.092806    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:29:27.093037 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.093022    2577 factory.go:153] Registering CRI-O factory
Sep 17 18:29:27.093092 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.093041    2577 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:29:27.093135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.093093    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:29:27.093135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.093115    2577 factory.go:103] Registering Raw factory
Sep 17 18:29:27.093135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.093131    2577 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:29:27.093584 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.093571    2577 manager.go:349] Starting recovery of all containers
Sep 17 18:29:27.094293 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.094272    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:29:27.094397 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.094380    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:29:27.097997 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.097972    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-667kx"
Sep 17 18:29:27.116463 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.116436    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-667kx"
Sep 17 18:29:27.121124 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.121103    2577 manager.go:354] Recovery completed
Sep 17 18:29:27.125863 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.125849    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:27.127642 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.127626    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:27.127712 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.127654    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:27.127712 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.127664    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:27.128158 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.128141    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:29:27.128158 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.128156    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:29:27.128265 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.128170    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:27.128265 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.128149    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:27.128987 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.128973    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:27.129040 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.128998    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:27.129040 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.129008    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:27.129489 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.129459    2577 policy_none.go:49] "None policy: Start"
Sep 17 18:29:27.129527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.129494    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:29:27.129527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.129504    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:29:27.130541 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.130531    2577 policy_none.go:47] "Start"
Sep 17 18:29:27.165445 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.165432    2577 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:29:27.178135 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.165470    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:29:27.178135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.165482    2577 server.go:87] "Starting device plugin registration server"
Sep 17 18:29:27.178135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.165702    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:29:27.178135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.165720    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:29:27.178135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.165825    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:29:27.178135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.165892    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:29:27.178135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.165911    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:29:27.178135 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.166265    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:29:27.178135 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.166313    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:27.231198 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.231174    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:29:27.232311 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.232294    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:29:27.232366 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.232342    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:29:27.232366 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.232365    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:29:27.232439 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.232407    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:29:27.234246 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.234227    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:27.266193 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.266144    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:27.266917 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.266890    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:27.266996 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.266937    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:27.266996 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.266947    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:27.266996 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.266969    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.272949 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.272934    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.273010 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.272955    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-128.ec2.internal\": node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:27.282239 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.282219    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:27.333218 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.333198    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal"]
Sep 17 18:29:27.333299 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.333256    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:27.334047 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.334031    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:27.334103 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.334060    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:27.334103 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.334070    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:27.335303 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.335291    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:27.335410 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.335398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.335456 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.335436    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:27.335955 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.335937    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:27.336043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.335966    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:27.336043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.335976    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:27.336043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.335940    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:27.336043 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.336041    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:27.336159 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.336052    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:27.337120 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.337107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.337162 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.337134    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:27.337751 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.337732    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:27.337751 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.337754    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:27.337863 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.337764    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:27.350434 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.350419    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-128.ec2.internal\" not found" node="ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.354246 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.354230    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-128.ec2.internal\" not found" node="ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.382656 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.382635    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:27.393748 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.393729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ecc6e94cce9716a0525faaf0d6791fbd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal\" (UID: \"ecc6e94cce9716a0525faaf0d6791fbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.393820 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.393751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecc6e94cce9716a0525faaf0d6791fbd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal\" (UID: \"ecc6e94cce9716a0525faaf0d6791fbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.393820 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.393767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/86685daa0de7a21d86d5b52a0666ae3d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-128.ec2.internal\" (UID: \"86685daa0de7a21d86d5b52a0666ae3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.483259 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.483237    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:27.494585 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.494565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ecc6e94cce9716a0525faaf0d6791fbd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal\" (UID: \"ecc6e94cce9716a0525faaf0d6791fbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.494663 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.494593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecc6e94cce9716a0525faaf0d6791fbd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal\" (UID: \"ecc6e94cce9716a0525faaf0d6791fbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.494663 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.494622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/86685daa0de7a21d86d5b52a0666ae3d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-128.ec2.internal\" (UID: \"86685daa0de7a21d86d5b52a0666ae3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.494663 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.494646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecc6e94cce9716a0525faaf0d6791fbd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal\" (UID: \"ecc6e94cce9716a0525faaf0d6791fbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.494663 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.494655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ecc6e94cce9716a0525faaf0d6791fbd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal\" (UID: \"ecc6e94cce9716a0525faaf0d6791fbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.494812 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.494668    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/86685daa0de7a21d86d5b52a0666ae3d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-128.ec2.internal\" (UID: \"86685daa0de7a21d86d5b52a0666ae3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.583689 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.583643    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:27.654063 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.654035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.657264 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:27.657247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:27.684587 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.684562    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:27.785063 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.785036    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:27.885644 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.885592    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:27.986067 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:27.986043    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-128.ec2.internal\" not found"
Sep 17 18:29:28.032573 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.032549    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:29:28.032957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.032677    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:28.069868 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.069708    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:28.070216 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.070203    2577 apiserver.go:52] "Watching apiserver"
Sep 17 18:29:28.078968 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.078946    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:29:28.079313 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.079291    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-kmgpj","openshift-multus/multus-additional-cni-plugins-xpp64","openshift-ovn-kubernetes/ovnkube-node-d5bqq","kube-system/konnectivity-agent-zn5q6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r","openshift-dns/node-resolver-pg2qm","openshift-image-registry/node-ca-x7n4t","openshift-multus/multus-sjv47","openshift-multus/network-metrics-daemon-mqx5t","openshift-network-diagnostics/network-check-target-cx897","openshift-network-operator/iptables-alerter-xgsr8","kube-system/global-pull-secret-syncer-j94pq"]
Sep 17 18:29:28.081042 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.081018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.082738 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.082574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.083664 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.083641    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:28.083810 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.083794    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gm2qz\""
Sep 17 18:29:28.084080 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.084061    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.084950 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.084834    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:28.086867 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.085426    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:29:28.086867 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.085627    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:29:28.086867 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.086222    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:29:28.086867 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.086229    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:29:28.086867 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.086513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p2729\""
Sep 17 18:29:28.086867 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.086554    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:29:28.087190 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.086922    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:29:28.087190 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.086969    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:29:28.088319 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.087522    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f8xjr\""
Sep 17 18:29:28.088319 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.087785    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:29:28.088319 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.088080    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:29:28.088517 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.088317    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:29:28.088737 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.088714    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:29:28.089062 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.088988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:28.091169 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.091152    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v565h\""
Sep 17 18:29:28.091405 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.091388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.091652 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.091633    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:29:28.091841 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.091828    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:29:28.092504 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.092491    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:28.092599 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.092491    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:28.092790 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.092773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.093474 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.093452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:29:28.093696 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.093676    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:29:28.093696 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.093695    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fdhmr\""
Sep 17 18:29:28.093874 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.093809    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:29:28.094503 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.094489    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.094992 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.094976    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:29:28.095101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.095056    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-slc89\""
Sep 17 18:29:28.095170 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.095105    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:29:28.095990 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.095973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.096759 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.096742    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:29:28.096829 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.096751    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:29:28.096829 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.096754    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:29:28.096964 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.096851    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dcp86\""
Sep 17 18:29:28.097925 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.097883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:28.098021 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.097926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.098021 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.097949    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:29:28.098021 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.097961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-socket-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.098021 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.097985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-sys-fs\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.098021 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.097982    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:28.098021 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-host\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czgbk\" (UniqueName: \"kubernetes.io/projected/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-kube-api-access-czgbk\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-kubelet\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-node-log\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50caee2e-986c-47b4-9198-ae60bbd7bd90-ovn-node-metrics-cert\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50caee2e-986c-47b4-9198-ae60bbd7bd90-ovnkube-script-lib\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93f04619-04fa-4035-8e10-92990294aa8f-konnectivity-ca\") pod \"konnectivity-agent-zn5q6\" (UID: \"93f04619-04fa-4035-8e10-92990294aa8f\") " pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-modprobe-d\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fffc0422-e6c2-4e7d-9c88-e6af27145389-tmp\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-slash\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-run-ovn\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-cni-bin\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.098406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-registration-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098410    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w2n8w\""
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-device-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmq4r\" (UniqueName: \"kubernetes.io/projected/a93c6cc9-d649-48f7-94fd-66df7247177c-kube-api-access-rmq4r\") pod \"node-resolver-pg2qm\" (UID: \"a93c6cc9-d649-48f7-94fd-66df7247177c\") " pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-sysconfig\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-cnibin\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-run-openvswitch\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-sysctl-conf\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-lib-modules\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-var-lib-kubelet\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-tuned\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93f04619-04fa-4035-8e10-92990294aa8f-agent-certs\") pod \"konnectivity-agent-zn5q6\" (UID: \"93f04619-04fa-4035-8e10-92990294aa8f\") " pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22c2\" (UniqueName: \"kubernetes.io/projected/b122bb58-8686-4cad-a65d-6c4c8ae5650e-kube-api-access-g22c2\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-sysctl-d\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.098957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a93c6cc9-d649-48f7-94fd-66df7247177c-hosts-file\") pod \"node-resolver-pg2qm\" (UID: \"a93c6cc9-d649-48f7-94fd-66df7247177c\") " pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93c6cc9-d649-48f7-94fd-66df7247177c-tmp-dir\") pod \"node-resolver-pg2qm\" (UID: \"a93c6cc9-d649-48f7-94fd-66df7247177c\") " pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-kubernetes\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-systemd\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.098980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-sys\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-system-cni-dir\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-var-lib-openvswitch\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50caee2e-986c-47b4-9198-ae60bbd7bd90-ovnkube-config\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-run\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-systemd-units\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-log-socket\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-cni-netd\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50caee2e-986c-47b4-9198-ae60bbd7bd90-env-overrides\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.099574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kcgk\" (UniqueName: \"kubernetes.io/projected/50caee2e-986c-47b4-9198-ae60bbd7bd90-kube-api-access-6kcgk\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.100113 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgzns\" (UniqueName: \"kubernetes.io/projected/fffc0422-e6c2-4e7d-9c88-e6af27145389-kube-api-access-jgzns\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.100113 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-run-netns\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.100113 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-run-systemd\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.100113 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-etc-openvswitch\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.100113 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-run-ovn-kubernetes\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.100113 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.099709    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:28.100113 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.099775    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:28.101144 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.101128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.102558 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.102543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:28.102622 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.102589    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:28.103640 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.103622    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:28.103697 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.103671    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:29:28.103697 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.103678    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-85fwd\""
Sep 17 18:29:28.103771 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.103699    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:28.104929 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.104916    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:28.104986 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.104979    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal"
Sep 17 18:29:28.105039 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.105025    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal"]
Sep 17 18:29:28.110703 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.110679    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:28.115751 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.115735    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:28.116064 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.116049    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal"]
Sep 17 18:29:28.123293 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.123251    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:24:27 +0000 UTC" deadline="2028-05-15 10:45:40.016942635 +0000 UTC"
Sep 17 18:29:28.123390 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.123293    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14536h16m11.893653755s"
Sep 17 18:29:28.126875 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.126856    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9lm98"
Sep 17 18:29:28.139031 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.139010    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9lm98"
Sep 17 18:29:28.149204 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.149187    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:28.193699 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.193683    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:29:28.199674 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.199755 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-host\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.199755 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fffc0422-e6c2-4e7d-9c88-e6af27145389-tmp\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.199755 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-kubelet\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.199755 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-node-log\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.199755 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50caee2e-986c-47b4-9198-ae60bbd7bd90-ovn-node-metrics-cert\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-kubelet\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-node-log\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50caee2e-986c-47b4-9198-ae60bbd7bd90-ovnkube-script-lib\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93f04619-04fa-4035-8e10-92990294aa8f-konnectivity-ca\") pod \"konnectivity-agent-zn5q6\" (UID: \"93f04619-04fa-4035-8e10-92990294aa8f\") " pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-host\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-lib-modules\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e8f8317-5d67-48a6-81f8-4856de66a377-host\") pod \"node-ca-x7n4t\" (UID: \"8e8f8317-5d67-48a6-81f8-4856de66a377\") " pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4f408b3-9984-4983-90f5-38a4195a2c2f-cni-binary-copy\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-lib-modules\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-daemon-config\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.200009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.199989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btshj\" (UniqueName: \"kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj\") pod \"network-check-target-cx897\" (UID: \"21c07112-ca6a-4083-9fad-f0b5b5afb0ba\") " pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200022    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ebc60a41-a81e-404d-978f-102cc369618b-dbus\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-cnibin\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-sysctl-conf\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-cnibin\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93f04619-04fa-4035-8e10-92990294aa8f-agent-certs\") pod \"konnectivity-agent-zn5q6\" (UID: \"93f04619-04fa-4035-8e10-92990294aa8f\") " pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22c2\" (UniqueName: \"kubernetes.io/projected/b122bb58-8686-4cad-a65d-6c4c8ae5650e-kube-api-access-g22c2\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-sysctl-d\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-sysctl-conf\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-socket-dir-parent\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-hostroot\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93c6cc9-d649-48f7-94fd-66df7247177c-tmp-dir\") pod \"node-resolver-pg2qm\" (UID: \"a93c6cc9-d649-48f7-94fd-66df7247177c\") " pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.200589 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-sysctl-d\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-kubernetes\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e8f8317-5d67-48a6-81f8-4856de66a377-serviceca\") pod \"node-ca-x7n4t\" (UID: \"8e8f8317-5d67-48a6-81f8-4856de66a377\") " pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcfnn\" (UniqueName: \"kubernetes.io/projected/656e881f-1982-4ce2-a57c-299fb552772b-kube-api-access-kcfnn\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-run-k8s-cni-cncf-io\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93f04619-04fa-4035-8e10-92990294aa8f-konnectivity-ca\") pod \"konnectivity-agent-zn5q6\" (UID: \"93f04619-04fa-4035-8e10-92990294aa8f\") " pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ebc60a41-a81e-404d-978f-102cc369618b-kubelet-config\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50caee2e-986c-47b4-9198-ae60bbd7bd90-ovnkube-script-lib\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-system-cni-dir\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-var-lib-cni-bin\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-systemd-units\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-log-socket\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50caee2e-986c-47b4-9198-ae60bbd7bd90-env-overrides\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-cni-dir\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-run-netns\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-var-lib-cni-multus\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.201450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/299a3d68-caa2-4dd3-bd34-3f69bbceba8e-iptables-alerter-script\") pod \"iptables-alerter-xgsr8\" (UID: \"299a3d68-caa2-4dd3-bd34-3f69bbceba8e\") " pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-systemd-units\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-socket-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-sys-fs\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-log-socket\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66wh\" (UniqueName: \"kubernetes.io/projected/e4f408b3-9984-4983-90f5-38a4195a2c2f-kube-api-access-c66wh\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czgbk\" (UniqueName: \"kubernetes.io/projected/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-kube-api-access-czgbk\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-modprobe-d\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-conf-dir\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-slash\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-run-ovn\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-cni-bin\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-registration-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-device-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmq4r\" (UniqueName: \"kubernetes.io/projected/a93c6cc9-d649-48f7-94fd-66df7247177c-kube-api-access-rmq4r\") pod \"node-resolver-pg2qm\" (UID: \"a93c6cc9-d649-48f7-94fd-66df7247177c\") " pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.202253 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-sysconfig\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-run-openvswitch\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50caee2e-986c-47b4-9198-ae60bbd7bd90-env-overrides\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-var-lib-kubelet\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-tuned\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2cq\" (UniqueName: \"kubernetes.io/projected/8e8f8317-5d67-48a6-81f8-4856de66a377-kube-api-access-7n2cq\") pod \"node-ca-x7n4t\" (UID: \"8e8f8317-5d67-48a6-81f8-4856de66a377\") " pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/299a3d68-caa2-4dd3-bd34-3f69bbceba8e-host-slash\") pod \"iptables-alerter-xgsr8\" (UID: \"299a3d68-caa2-4dd3-bd34-3f69bbceba8e\") " pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-cni-bin\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a93c6cc9-d649-48f7-94fd-66df7247177c-hosts-file\") pod \"node-resolver-pg2qm\" (UID: \"a93c6cc9-d649-48f7-94fd-66df7247177c\") " pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-systemd\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-sys\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-var-lib-kubelet\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-run-multus-certs\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-etc-kubernetes\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkzg\" (UniqueName: \"kubernetes.io/projected/299a3d68-caa2-4dd3-bd34-3f69bbceba8e-kube-api-access-xvkzg\") pod \"iptables-alerter-xgsr8\" (UID: \"299a3d68-caa2-4dd3-bd34-3f69bbceba8e\") " pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-var-lib-openvswitch\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.203181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50caee2e-986c-47b4-9198-ae60bbd7bd90-ovnkube-config\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-run\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-modprobe-d\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-device-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-registration-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93c6cc9-d649-48f7-94fd-66df7247177c-tmp-dir\") pod \"node-resolver-pg2qm\" (UID: \"a93c6cc9-d649-48f7-94fd-66df7247177c\") " pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201774    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-run-ovn\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-slash\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-cnibin\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.200645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-system-cni-dir\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201845    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-sys-fs\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201851    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-kubernetes\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a93c6cc9-d649-48f7-94fd-66df7247177c-hosts-file\") pod \"node-resolver-pg2qm\" (UID: \"a93c6cc9-d649-48f7-94fd-66df7247177c\") " pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-var-lib-kubelet\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.204025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-cni-netd\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.201993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-run\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kcgk\" (UniqueName: \"kubernetes.io/projected/50caee2e-986c-47b4-9198-ae60bbd7bd90-kube-api-access-6kcgk\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b122bb58-8686-4cad-a65d-6c4c8ae5650e-socket-dir\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzns\" (UniqueName: \"kubernetes.io/projected/fffc0422-e6c2-4e7d-9c88-e6af27145389-kube-api-access-jgzns\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-run-openvswitch\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-sys\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-sysconfig\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-system-cni-dir\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-run-netns\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-run-systemd\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-etc-openvswitch\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-run-ovn-kubernetes\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-run-ovn-kubernetes\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-run-netns\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50caee2e-986c-47b4-9198-ae60bbd7bd90-ovnkube-config\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.204578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-run-systemd\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.205060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-var-lib-openvswitch\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.205060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-etc-openvswitch\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.205060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-cni-netd\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.205060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-systemd\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.205060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50caee2e-986c-47b4-9198-ae60bbd7bd90-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.205060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.202540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.205060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.203846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50caee2e-986c-47b4-9198-ae60bbd7bd90-ovn-node-metrics-cert\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.205060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.204043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93f04619-04fa-4035-8e10-92990294aa8f-agent-certs\") pod \"konnectivity-agent-zn5q6\" (UID: \"93f04619-04fa-4035-8e10-92990294aa8f\") " pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:28.205060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.204277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fffc0422-e6c2-4e7d-9c88-e6af27145389-tmp\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.205060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.204332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fffc0422-e6c2-4e7d-9c88-e6af27145389-etc-tuned\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.208693 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.208669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czgbk\" (UniqueName: \"kubernetes.io/projected/0833754b-b18a-43ba-bb8d-c5f1c4591a7a-kube-api-access-czgbk\") pod \"multus-additional-cni-plugins-xpp64\" (UID: \"0833754b-b18a-43ba-bb8d-c5f1c4591a7a\") " pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.208792 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.208770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22c2\" (UniqueName: \"kubernetes.io/projected/b122bb58-8686-4cad-a65d-6c4c8ae5650e-kube-api-access-g22c2\") pod \"aws-ebs-csi-driver-node-ltk5r\" (UID: \"b122bb58-8686-4cad-a65d-6c4c8ae5650e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.208962 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.208944    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmq4r\" (UniqueName: \"kubernetes.io/projected/a93c6cc9-d649-48f7-94fd-66df7247177c-kube-api-access-rmq4r\") pod \"node-resolver-pg2qm\" (UID: \"a93c6cc9-d649-48f7-94fd-66df7247177c\") " pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.212441 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.212423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kcgk\" (UniqueName: \"kubernetes.io/projected/50caee2e-986c-47b4-9198-ae60bbd7bd90-kube-api-access-6kcgk\") pod \"ovnkube-node-d5bqq\" (UID: \"50caee2e-986c-47b4-9198-ae60bbd7bd90\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.214002 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.213983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgzns\" (UniqueName: \"kubernetes.io/projected/fffc0422-e6c2-4e7d-9c88-e6af27145389-kube-api-access-jgzns\") pod \"tuned-kmgpj\" (UID: \"fffc0422-e6c2-4e7d-9c88-e6af27145389\") " pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.266646 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.266622    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86685daa0de7a21d86d5b52a0666ae3d.slice/crio-eb4c325271a582e2b8c7e2f4f2303b7b1e142bd232cd115623c65473d800fbbf WatchSource:0}: Error finding container eb4c325271a582e2b8c7e2f4f2303b7b1e142bd232cd115623c65473d800fbbf: Status 404 returned error can't find the container with id eb4c325271a582e2b8c7e2f4f2303b7b1e142bd232cd115623c65473d800fbbf
Sep 17 18:29:28.267134 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.267112    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc6e94cce9716a0525faaf0d6791fbd.slice/crio-5c839162e925259461aa7686f64d9b7f724e81e8f6f6d41c06eca18bcfc94072 WatchSource:0}: Error finding container 5c839162e925259461aa7686f64d9b7f724e81e8f6f6d41c06eca18bcfc94072: Status 404 returned error can't find the container with id 5c839162e925259461aa7686f64d9b7f724e81e8f6f6d41c06eca18bcfc94072
Sep 17 18:29:28.271807 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.271793    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:29:28.302913 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.302883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e8f8317-5d67-48a6-81f8-4856de66a377-host\") pod \"node-ca-x7n4t\" (UID: \"8e8f8317-5d67-48a6-81f8-4856de66a377\") " pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.302980 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.302925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4f408b3-9984-4983-90f5-38a4195a2c2f-cni-binary-copy\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.302980 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.302946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-daemon-config\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.302980 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.302968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btshj\" (UniqueName: \"kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj\") pod \"network-check-target-cx897\" (UID: \"21c07112-ca6a-4083-9fad-f0b5b5afb0ba\") " pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:28.303118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.302985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e8f8317-5d67-48a6-81f8-4856de66a377-host\") pod \"node-ca-x7n4t\" (UID: \"8e8f8317-5d67-48a6-81f8-4856de66a377\") " pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.303118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.302992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ebc60a41-a81e-404d-978f-102cc369618b-dbus\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:28.303118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:28.303118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-socket-dir-parent\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-hostroot\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e8f8317-5d67-48a6-81f8-4856de66a377-serviceca\") pod \"node-ca-x7n4t\" (UID: \"8e8f8317-5d67-48a6-81f8-4856de66a377\") " pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.303118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfnn\" (UniqueName: \"kubernetes.io/projected/656e881f-1982-4ce2-a57c-299fb552772b-kube-api-access-kcfnn\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:28.303118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-run-k8s-cni-cncf-io\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-socket-dir-parent\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ebc60a41-a81e-404d-978f-102cc369618b-kubelet-config\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.303136    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-var-lib-cni-bin\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-cni-dir\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.303203    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs podName:656e881f-1982-4ce2-a57c-299fb552772b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:28.803180251 +0000 UTC m=+3.849583148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs") pod "network-metrics-daemon-mqx5t" (UID: "656e881f-1982-4ce2-a57c-299fb552772b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ebc60a41-a81e-404d-978f-102cc369618b-dbus\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-hostroot\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-run-netns\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-var-lib-cni-bin\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-var-lib-cni-multus\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ebc60a41-a81e-404d-978f-102cc369618b-kubelet-config\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-run-k8s-cni-cncf-io\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-cni-dir\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-run-netns\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/299a3d68-caa2-4dd3-bd34-3f69bbceba8e-iptables-alerter-script\") pod \"iptables-alerter-xgsr8\" (UID: \"299a3d68-caa2-4dd3-bd34-3f69bbceba8e\") " pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-var-lib-cni-multus\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66wh\" (UniqueName: \"kubernetes.io/projected/e4f408b3-9984-4983-90f5-38a4195a2c2f-kube-api-access-c66wh\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.303527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-conf-dir\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2cq\" (UniqueName: \"kubernetes.io/projected/8e8f8317-5d67-48a6-81f8-4856de66a377-kube-api-access-7n2cq\") pod \"node-ca-x7n4t\" (UID: \"8e8f8317-5d67-48a6-81f8-4856de66a377\") " pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-conf-dir\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/299a3d68-caa2-4dd3-bd34-3f69bbceba8e-host-slash\") pod \"iptables-alerter-xgsr8\" (UID: \"299a3d68-caa2-4dd3-bd34-3f69bbceba8e\") " pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-var-lib-kubelet\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4f408b3-9984-4983-90f5-38a4195a2c2f-cni-binary-copy\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e8f8317-5d67-48a6-81f8-4856de66a377-serviceca\") pod \"node-ca-x7n4t\" (UID: \"8e8f8317-5d67-48a6-81f8-4856de66a377\") " pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-run-multus-certs\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-run-multus-certs\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-etc-kubernetes\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-host-var-lib-kubelet\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/299a3d68-caa2-4dd3-bd34-3f69bbceba8e-host-slash\") pod \"iptables-alerter-xgsr8\" (UID: \"299a3d68-caa2-4dd3-bd34-3f69bbceba8e\") " pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkzg\" (UniqueName: \"kubernetes.io/projected/299a3d68-caa2-4dd3-bd34-3f69bbceba8e-kube-api-access-xvkzg\") pod \"iptables-alerter-xgsr8\" (UID: \"299a3d68-caa2-4dd3-bd34-3f69bbceba8e\") " pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-cnibin\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-etc-kubernetes\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-system-cni-dir\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303719    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-cnibin\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.303770    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:28.304165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4f408b3-9984-4983-90f5-38a4195a2c2f-system-cni-dir\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304699 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4f408b3-9984-4983-90f5-38a4195a2c2f-multus-daemon-config\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.304699 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.303811    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/299a3d68-caa2-4dd3-bd34-3f69bbceba8e-iptables-alerter-script\") pod \"iptables-alerter-xgsr8\" (UID: \"299a3d68-caa2-4dd3-bd34-3f69bbceba8e\") " pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.304699 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.303818    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret podName:ebc60a41-a81e-404d-978f-102cc369618b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:28.803802526 +0000 UTC m=+3.850205421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret") pod "global-pull-secret-syncer-j94pq" (UID: "ebc60a41-a81e-404d-978f-102cc369618b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:28.310388 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.310368    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:28.310388 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.310390    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:28.310524 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.310402    2577 projected.go:196] Error preparing data for projected volume kube-api-access-btshj for pod openshift-network-diagnostics/network-check-target-cx897: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:28.310524 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.310454    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj podName:21c07112-ca6a-4083-9fad-f0b5b5afb0ba nodeName:}" failed. No retries permitted until 2026-09-17 18:29:28.810441097 +0000 UTC m=+3.856843977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-btshj" (UniqueName: "kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj") pod "network-check-target-cx897" (UID: "21c07112-ca6a-4083-9fad-f0b5b5afb0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:28.312045 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.312022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkzg\" (UniqueName: \"kubernetes.io/projected/299a3d68-caa2-4dd3-bd34-3f69bbceba8e-kube-api-access-xvkzg\") pod \"iptables-alerter-xgsr8\" (UID: \"299a3d68-caa2-4dd3-bd34-3f69bbceba8e\") " pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.312133 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.312029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2cq\" (UniqueName: \"kubernetes.io/projected/8e8f8317-5d67-48a6-81f8-4856de66a377-kube-api-access-7n2cq\") pod \"node-ca-x7n4t\" (UID: \"8e8f8317-5d67-48a6-81f8-4856de66a377\") " pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.312505 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.312488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfnn\" (UniqueName: \"kubernetes.io/projected/656e881f-1982-4ce2-a57c-299fb552772b-kube-api-access-kcfnn\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:28.313059 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.313044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66wh\" (UniqueName: \"kubernetes.io/projected/e4f408b3-9984-4983-90f5-38a4195a2c2f-kube-api-access-c66wh\") pod \"multus-sjv47\" (UID: \"e4f408b3-9984-4983-90f5-38a4195a2c2f\") " pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.415861 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.415814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kmgpj"
Sep 17 18:29:28.423236 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.423215    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffc0422_e6c2_4e7d_9c88_e6af27145389.slice/crio-11edcfee6d92d42419a25a056f8554bb5c5df6f3fca38ac37a289e5c22a06cfc WatchSource:0}: Error finding container 11edcfee6d92d42419a25a056f8554bb5c5df6f3fca38ac37a289e5c22a06cfc: Status 404 returned error can't find the container with id 11edcfee6d92d42419a25a056f8554bb5c5df6f3fca38ac37a289e5c22a06cfc
Sep 17 18:29:28.430842 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.430826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xpp64"
Sep 17 18:29:28.435773 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.435754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:28.438019 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.438000    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0833754b_b18a_43ba_bb8d_c5f1c4591a7a.slice/crio-3440e78350561e7835959a521ee8b75ca9a32b2e689f4ead64f75b8c0f5edcf0 WatchSource:0}: Error finding container 3440e78350561e7835959a521ee8b75ca9a32b2e689f4ead64f75b8c0f5edcf0: Status 404 returned error can't find the container with id 3440e78350561e7835959a521ee8b75ca9a32b2e689f4ead64f75b8c0f5edcf0
Sep 17 18:29:28.442768 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.442750    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50caee2e_986c_47b4_9198_ae60bbd7bd90.slice/crio-4a683a2f721cedb8a1d04a0723007e10ac6792f9465909913483cd000150d691 WatchSource:0}: Error finding container 4a683a2f721cedb8a1d04a0723007e10ac6792f9465909913483cd000150d691: Status 404 returned error can't find the container with id 4a683a2f721cedb8a1d04a0723007e10ac6792f9465909913483cd000150d691
Sep 17 18:29:28.448205 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.448185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:28.455152 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.455129    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f04619_04fa_4035_8e10_92990294aa8f.slice/crio-1aa96270cd010b3492f3d786fa476f970183ce64df1a64b7e41464d50a5ea4d9 WatchSource:0}: Error finding container 1aa96270cd010b3492f3d786fa476f970183ce64df1a64b7e41464d50a5ea4d9: Status 404 returned error can't find the container with id 1aa96270cd010b3492f3d786fa476f970183ce64df1a64b7e41464d50a5ea4d9
Sep 17 18:29:28.465968 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.465952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r"
Sep 17 18:29:28.472233 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.472214    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb122bb58_8686_4cad_a65d_6c4c8ae5650e.slice/crio-7295f66f7c47a1a17399251a9f3d7e2c0ea765a4e3e52edfbc512714bf0edac0 WatchSource:0}: Error finding container 7295f66f7c47a1a17399251a9f3d7e2c0ea765a4e3e52edfbc512714bf0edac0: Status 404 returned error can't find the container with id 7295f66f7c47a1a17399251a9f3d7e2c0ea765a4e3e52edfbc512714bf0edac0
Sep 17 18:29:28.482458 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.482443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pg2qm"
Sep 17 18:29:28.488194 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.488174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x7n4t"
Sep 17 18:29:28.488872 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.488856    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93c6cc9_d649_48f7_94fd_66df7247177c.slice/crio-7696f1c188bceddb6a8485c594edf73698c580649be34f916ddb541ce4bc17fe WatchSource:0}: Error finding container 7696f1c188bceddb6a8485c594edf73698c580649be34f916ddb541ce4bc17fe: Status 404 returned error can't find the container with id 7696f1c188bceddb6a8485c594edf73698c580649be34f916ddb541ce4bc17fe
Sep 17 18:29:28.494734 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.494714    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8f8317_5d67_48a6_81f8_4856de66a377.slice/crio-91e4a040fd7cc6dd3bdafea0b83c91b79b80492ef1f515602a5dc228fd2fc721 WatchSource:0}: Error finding container 91e4a040fd7cc6dd3bdafea0b83c91b79b80492ef1f515602a5dc228fd2fc721: Status 404 returned error can't find the container with id 91e4a040fd7cc6dd3bdafea0b83c91b79b80492ef1f515602a5dc228fd2fc721
Sep 17 18:29:28.502935 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.502919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sjv47"
Sep 17 18:29:28.509801 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.509778    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f408b3_9984_4983_90f5_38a4195a2c2f.slice/crio-e1b1dfa56b51f986de803c8191528f2cae7d4dff74368726030fec8271011311 WatchSource:0}: Error finding container e1b1dfa56b51f986de803c8191528f2cae7d4dff74368726030fec8271011311: Status 404 returned error can't find the container with id e1b1dfa56b51f986de803c8191528f2cae7d4dff74368726030fec8271011311
Sep 17 18:29:28.515651 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.515636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xgsr8"
Sep 17 18:29:28.522071 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:29:28.522046    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299a3d68_caa2_4dd3_bd34_3f69bbceba8e.slice/crio-b41fe1836406d17de2384efd50ff3b99a3d21555136db58578b8c4a912693d60 WatchSource:0}: Error finding container b41fe1836406d17de2384efd50ff3b99a3d21555136db58578b8c4a912693d60: Status 404 returned error can't find the container with id b41fe1836406d17de2384efd50ff3b99a3d21555136db58578b8c4a912693d60
Sep 17 18:29:28.684136 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.684072    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:28.807153 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.807115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:28.807297 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.807189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:28.807354 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.807319    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:28.807423 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.807373    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs podName:656e881f-1982-4ce2-a57c-299fb552772b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:29.807355819 +0000 UTC m=+4.853758705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs") pod "network-metrics-daemon-mqx5t" (UID: "656e881f-1982-4ce2-a57c-299fb552772b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:28.807766 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.807746    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:28.807850 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.807805    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret podName:ebc60a41-a81e-404d-978f-102cc369618b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:29.807788818 +0000 UTC m=+4.854191703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret") pod "global-pull-secret-syncer-j94pq" (UID: "ebc60a41-a81e-404d-978f-102cc369618b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:28.907897 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:28.907864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btshj\" (UniqueName: \"kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj\") pod \"network-check-target-cx897\" (UID: \"21c07112-ca6a-4083-9fad-f0b5b5afb0ba\") " pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:28.908095 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.908074    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:28.908161 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.908098    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:28.908161 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.908111    2577 projected.go:196] Error preparing data for projected volume kube-api-access-btshj for pod openshift-network-diagnostics/network-check-target-cx897: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:28.908262 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:28.908167    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj podName:21c07112-ca6a-4083-9fad-f0b5b5afb0ba nodeName:}" failed. No retries permitted until 2026-09-17 18:29:29.90814848 +0000 UTC m=+4.954551365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-btshj" (UniqueName: "kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj") pod "network-check-target-cx897" (UID: "21c07112-ca6a-4083-9fad-f0b5b5afb0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:29.021034 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.020788    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:29.143279 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.143165    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:28 +0000 UTC" deadline="2028-05-18 12:40:51.642484091 +0000 UTC"
Sep 17 18:29:29.143279 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.143236    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14610h11m22.499251553s"
Sep 17 18:29:29.268970 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.268775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xgsr8" event={"ID":"299a3d68-caa2-4dd3-bd34-3f69bbceba8e","Type":"ContainerStarted","Data":"b41fe1836406d17de2384efd50ff3b99a3d21555136db58578b8c4a912693d60"}
Sep 17 18:29:29.288348 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.288278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sjv47" event={"ID":"e4f408b3-9984-4983-90f5-38a4195a2c2f","Type":"ContainerStarted","Data":"e1b1dfa56b51f986de803c8191528f2cae7d4dff74368726030fec8271011311"}
Sep 17 18:29:29.322500 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.322468    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq" event={"ID":"50caee2e-986c-47b4-9198-ae60bbd7bd90","Type":"ContainerStarted","Data":"4a683a2f721cedb8a1d04a0723007e10ac6792f9465909913483cd000150d691"}
Sep 17 18:29:29.339189 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.339159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpp64" event={"ID":"0833754b-b18a-43ba-bb8d-c5f1c4591a7a","Type":"ContainerStarted","Data":"3440e78350561e7835959a521ee8b75ca9a32b2e689f4ead64f75b8c0f5edcf0"}
Sep 17 18:29:29.358918 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.358880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal" event={"ID":"86685daa0de7a21d86d5b52a0666ae3d","Type":"ContainerStarted","Data":"eb4c325271a582e2b8c7e2f4f2303b7b1e142bd232cd115623c65473d800fbbf"}
Sep 17 18:29:29.366944 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.364387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x7n4t" event={"ID":"8e8f8317-5d67-48a6-81f8-4856de66a377","Type":"ContainerStarted","Data":"91e4a040fd7cc6dd3bdafea0b83c91b79b80492ef1f515602a5dc228fd2fc721"}
Sep 17 18:29:29.370612 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.370586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pg2qm" event={"ID":"a93c6cc9-d649-48f7-94fd-66df7247177c","Type":"ContainerStarted","Data":"7696f1c188bceddb6a8485c594edf73698c580649be34f916ddb541ce4bc17fe"}
Sep 17 18:29:29.395144 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.395114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r" event={"ID":"b122bb58-8686-4cad-a65d-6c4c8ae5650e","Type":"ContainerStarted","Data":"7295f66f7c47a1a17399251a9f3d7e2c0ea765a4e3e52edfbc512714bf0edac0"}
Sep 17 18:29:29.414665 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.414310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zn5q6" event={"ID":"93f04619-04fa-4035-8e10-92990294aa8f","Type":"ContainerStarted","Data":"1aa96270cd010b3492f3d786fa476f970183ce64df1a64b7e41464d50a5ea4d9"}
Sep 17 18:29:29.423531 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.423506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kmgpj" event={"ID":"fffc0422-e6c2-4e7d-9c88-e6af27145389","Type":"ContainerStarted","Data":"11edcfee6d92d42419a25a056f8554bb5c5df6f3fca38ac37a289e5c22a06cfc"}
Sep 17 18:29:29.439977 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.439888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal" event={"ID":"ecc6e94cce9716a0525faaf0d6791fbd","Type":"ContainerStarted","Data":"5c839162e925259461aa7686f64d9b7f724e81e8f6f6d41c06eca18bcfc94072"}
Sep 17 18:29:29.819928 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.819127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:29.819928 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.819204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:29.819928 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:29.819332    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:29.819928 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:29.819395    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs podName:656e881f-1982-4ce2-a57c-299fb552772b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:31.81937556 +0000 UTC m=+6.865778441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs") pod "network-metrics-daemon-mqx5t" (UID: "656e881f-1982-4ce2-a57c-299fb552772b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:29.819928 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:29.819824    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:29.819928 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:29.819877    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret podName:ebc60a41-a81e-404d-978f-102cc369618b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:31.819861273 +0000 UTC m=+6.866264157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret") pod "global-pull-secret-syncer-j94pq" (UID: "ebc60a41-a81e-404d-978f-102cc369618b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:29.920396 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:29.920356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btshj\" (UniqueName: \"kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj\") pod \"network-check-target-cx897\" (UID: \"21c07112-ca6a-4083-9fad-f0b5b5afb0ba\") " pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:29.920591 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:29.920572    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:29.920661 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:29.920599    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:29.920661 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:29.920611    2577 projected.go:196] Error preparing data for projected volume kube-api-access-btshj for pod openshift-network-diagnostics/network-check-target-cx897: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:29.920761 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:29.920666    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj podName:21c07112-ca6a-4083-9fad-f0b5b5afb0ba nodeName:}" failed. No retries permitted until 2026-09-17 18:29:31.920649326 +0000 UTC m=+6.967052212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-btshj" (UniqueName: "kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj") pod "network-check-target-cx897" (UID: "21c07112-ca6a-4083-9fad-f0b5b5afb0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:30.143554 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:30.143466    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:28 +0000 UTC" deadline="2028-03-26 09:11:17.181290449 +0000 UTC"
Sep 17 18:29:30.143554 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:30.143507    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13334h41m47.037788171s"
Sep 17 18:29:30.232998 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:30.232966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:30.233169 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:30.233101    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:30.233231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:30.233188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:30.233717 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:30.233274    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:30.233717 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:30.233336    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:30.233717 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:30.233402    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:31.835866 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:31.835329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:31.835866 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:31.835408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:31.835866 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:31.835472    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:31.835866 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:31.835524    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:31.835866 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:31.835539    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret podName:ebc60a41-a81e-404d-978f-102cc369618b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:35.835521382 +0000 UTC m=+10.881924265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret") pod "global-pull-secret-syncer-j94pq" (UID: "ebc60a41-a81e-404d-978f-102cc369618b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:31.835866 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:31.835580    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs podName:656e881f-1982-4ce2-a57c-299fb552772b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:35.835563697 +0000 UTC m=+10.881966581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs") pod "network-metrics-daemon-mqx5t" (UID: "656e881f-1982-4ce2-a57c-299fb552772b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:31.936546 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:31.936453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btshj\" (UniqueName: \"kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj\") pod \"network-check-target-cx897\" (UID: \"21c07112-ca6a-4083-9fad-f0b5b5afb0ba\") " pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:31.936712 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:31.936634    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:31.936712 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:31.936663    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:31.936712 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:31.936677    2577 projected.go:196] Error preparing data for projected volume kube-api-access-btshj for pod openshift-network-diagnostics/network-check-target-cx897: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:31.936873 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:31.936734    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj podName:21c07112-ca6a-4083-9fad-f0b5b5afb0ba nodeName:}" failed. No retries permitted until 2026-09-17 18:29:35.936715759 +0000 UTC m=+10.983118644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-btshj" (UniqueName: "kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj") pod "network-check-target-cx897" (UID: "21c07112-ca6a-4083-9fad-f0b5b5afb0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:32.233539 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:32.233460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:32.233539 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:32.233487    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:32.233717 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:32.233463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:32.233717 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:32.233584    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:32.233717 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:32.233644    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:32.233717 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:32.233709    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:34.233101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:34.233003    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:34.233101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:34.233028    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:34.233101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:34.233028    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:34.233682 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:34.233142    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:34.233682 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:34.233406    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:34.233682 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:34.233486    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:35.869803 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:35.869728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:35.870299 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:35.869886    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:35.870299 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:35.869918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:35.870299 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:35.869974    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret podName:ebc60a41-a81e-404d-978f-102cc369618b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.869954451 +0000 UTC m=+18.916357336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret") pod "global-pull-secret-syncer-j94pq" (UID: "ebc60a41-a81e-404d-978f-102cc369618b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:35.870299 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:35.870037    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:35.870299 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:35.870091    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs podName:656e881f-1982-4ce2-a57c-299fb552772b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.870075133 +0000 UTC m=+18.916478027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs") pod "network-metrics-daemon-mqx5t" (UID: "656e881f-1982-4ce2-a57c-299fb552772b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:35.970344 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:35.970280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btshj\" (UniqueName: \"kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj\") pod \"network-check-target-cx897\" (UID: \"21c07112-ca6a-4083-9fad-f0b5b5afb0ba\") " pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:35.970507 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:35.970470    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:35.970507 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:35.970498    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:35.970611 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:35.970511    2577 projected.go:196] Error preparing data for projected volume kube-api-access-btshj for pod openshift-network-diagnostics/network-check-target-cx897: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:35.970611 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:35.970574    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj podName:21c07112-ca6a-4083-9fad-f0b5b5afb0ba nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.970556167 +0000 UTC m=+19.016959053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-btshj" (UniqueName: "kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj") pod "network-check-target-cx897" (UID: "21c07112-ca6a-4083-9fad-f0b5b5afb0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:36.233229 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:36.233071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:36.233229 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:36.233089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:36.233229 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:36.233216    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:36.233510 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:36.233269    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:36.233510 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:36.233289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:36.233510 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:36.233382    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:38.232801 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:38.232767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:38.233262 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:38.232881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:38.233262 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:38.232891    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:38.233262 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:38.232931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:38.233262 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:38.233003    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:38.233262 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:38.233070    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:40.233277 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:40.233246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:40.233680 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:40.233279    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:40.233680 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:40.233352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:40.233680 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:40.233371    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:40.233680 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:40.233428    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:40.233680 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:40.233528    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:42.233460 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:42.233425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:42.233870 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:42.233538    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:42.233870 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:42.233559    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:42.233870 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:42.233549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:42.233870 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:42.233639    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:42.233870 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:42.233716    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:43.933451 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:43.933407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:43.934085 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:43.933492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:43.934085 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:43.933594    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:43.934085 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:43.933671    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs podName:656e881f-1982-4ce2-a57c-299fb552772b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.93365252 +0000 UTC m=+34.980055406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs") pod "network-metrics-daemon-mqx5t" (UID: "656e881f-1982-4ce2-a57c-299fb552772b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:43.934085 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:43.933599    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:43.934085 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:43.933768    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret podName:ebc60a41-a81e-404d-978f-102cc369618b nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.933748412 +0000 UTC m=+34.980151307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret") pod "global-pull-secret-syncer-j94pq" (UID: "ebc60a41-a81e-404d-978f-102cc369618b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:44.033988 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:44.033961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btshj\" (UniqueName: \"kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj\") pod \"network-check-target-cx897\" (UID: \"21c07112-ca6a-4083-9fad-f0b5b5afb0ba\") " pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:44.034162 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:44.034139    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:44.034247 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:44.034171    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:44.034247 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:44.034186    2577 projected.go:196] Error preparing data for projected volume kube-api-access-btshj for pod openshift-network-diagnostics/network-check-target-cx897: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:44.034328 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:44.034262    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj podName:21c07112-ca6a-4083-9fad-f0b5b5afb0ba nodeName:}" failed. No retries permitted until 2026-09-17 18:30:00.034243669 +0000 UTC m=+35.080646553 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-btshj" (UniqueName: "kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj") pod "network-check-target-cx897" (UID: "21c07112-ca6a-4083-9fad-f0b5b5afb0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:44.232957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:44.232873    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:44.232957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:44.232863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:44.233234 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:44.232874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:44.233234 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:44.233011    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:44.233234 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:44.233088    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:44.233234 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:44.233145    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:46.233227 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:46.233190    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:46.233227 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:46.233224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:46.233739 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:46.233191    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:46.233739 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:46.233321    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:46.233739 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:46.233420    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:46.233739 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:46.233539    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:47.479258 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.479216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r" event={"ID":"b122bb58-8686-4cad-a65d-6c4c8ae5650e","Type":"ContainerStarted","Data":"1bb8960a227bb01b0004feee401f887e090da6f3b6b6ed479b272e802ee75cbd"}
Sep 17 18:29:47.480730 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.480703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zn5q6" event={"ID":"93f04619-04fa-4035-8e10-92990294aa8f","Type":"ContainerStarted","Data":"ff62fc509d0bdd8d086151a0ea6dd1f7544d77c3448c27115d915c63cbf90168"}
Sep 17 18:29:47.482897 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.482816    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kmgpj" event={"ID":"fffc0422-e6c2-4e7d-9c88-e6af27145389","Type":"ContainerStarted","Data":"415ad5c84a1ebf71cfed46279274f662f41ab004e721c1b5d8ec1cc2e5c77024"}
Sep 17 18:29:47.484739 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.484670    2577 generic.go:356] "Generic (PLEG): container finished" podID="ecc6e94cce9716a0525faaf0d6791fbd" containerID="5a9e26b1f617e332f3067e5d18a01b3bca5e07bf9d6e92226ead9e47bad6f364" exitCode=0
Sep 17 18:29:47.484739 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.484695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal" event={"ID":"ecc6e94cce9716a0525faaf0d6791fbd","Type":"ContainerDied","Data":"5a9e26b1f617e332f3067e5d18a01b3bca5e07bf9d6e92226ead9e47bad6f364"}
Sep 17 18:29:47.487447 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.487425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sjv47" event={"ID":"e4f408b3-9984-4983-90f5-38a4195a2c2f","Type":"ContainerStarted","Data":"3bbfacc883ff5cb0ba4710b7b2fd559012ec4dd70f68bbf8071ce5bbbc4ad2e9"}
Sep 17 18:29:47.489897 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.489878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq" event={"ID":"50caee2e-986c-47b4-9198-ae60bbd7bd90","Type":"ContainerStarted","Data":"2d8b16418f8e1256ea97c4673578b8ac45b565c13719bd15a8fee3c4579e7e87"}
Sep 17 18:29:47.490024 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.489968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq" event={"ID":"50caee2e-986c-47b4-9198-ae60bbd7bd90","Type":"ContainerStarted","Data":"a9aef0efb636fd20844f5e63bfeecbb3500534b49acfcc0ad8569403a7fc8550"}
Sep 17 18:29:47.490072 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.490044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq" event={"ID":"50caee2e-986c-47b4-9198-ae60bbd7bd90","Type":"ContainerStarted","Data":"40e3b30235c7b9e9fe53102652970f4b1d89094ae67fbfdea72ef08fe7860042"}
Sep 17 18:29:47.490116 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.490102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq" event={"ID":"50caee2e-986c-47b4-9198-ae60bbd7bd90","Type":"ContainerStarted","Data":"3f4026ec443c0d203d211d99e0b9c728e7ab8cd1d5cec10f3a5511e80de1a8ed"}
Sep 17 18:29:47.491413 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.491391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpp64" event={"ID":"0833754b-b18a-43ba-bb8d-c5f1c4591a7a","Type":"ContainerStarted","Data":"6de0095b54ca5cb2a107d8d57061a45b5750a7f9595277250d3e1cc5ff801a70"}
Sep 17 18:29:47.492819 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.492796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal" event={"ID":"86685daa0de7a21d86d5b52a0666ae3d","Type":"ContainerStarted","Data":"7cabfb87c4674a810e7decaf3af38804b16a888d47893a24513dabb63765724d"}
Sep 17 18:29:47.494336 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.494318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x7n4t" event={"ID":"8e8f8317-5d67-48a6-81f8-4856de66a377","Type":"ContainerStarted","Data":"3f47296932ecf272810ed8c38d278e7943ad5a2aebef6dcaa0e19372d143f91f"}
Sep 17 18:29:47.495816 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.495781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pg2qm" event={"ID":"a93c6cc9-d649-48f7-94fd-66df7247177c","Type":"ContainerStarted","Data":"2b7684fef1280eccbb6a5935eb4e37926f9e05596f2938e9c98269879d1a7fca"}
Sep 17 18:29:47.496236 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.496197    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zn5q6" podStartSLOduration=1.962870935 podStartE2EDuration="20.496184837s" totalImagesPullingTime="18.533313902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:47.495728487 +0000 UTC m=+22.542131402" watchObservedRunningTime="2026-09-17 18:29:47.496184837 +0000 UTC m=+22.542587738"
Sep 17 18:29:47.510032 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.509996    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x7n4t" podStartSLOduration=1.992913437 podStartE2EDuration="20.50998394s" totalImagesPullingTime="18.517070503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:47.509530634 +0000 UTC m=+22.555933560" watchObservedRunningTime="2026-09-17 18:29:47.50998394 +0000 UTC m=+22.556386842"
Sep 17 18:29:47.525559 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.525523    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sjv47" podStartSLOduration=1.965488787 podStartE2EDuration="20.525513286s" totalImagesPullingTime="18.560024499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:47.525347739 +0000 UTC m=+22.571750641" watchObservedRunningTime="2026-09-17 18:29:47.525513286 +0000 UTC m=+22.571916187"
Sep 17 18:29:47.551321 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.551301    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:47.551805 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.551788    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:47.560296 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.560242    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-128.ec2.internal" podStartSLOduration=19.560226475 podStartE2EDuration="19.560226475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:47.559715143 +0000 UTC m=+22.606118045" watchObservedRunningTime="2026-09-17 18:29:47.560226475 +0000 UTC m=+22.606629373"
Sep 17 18:29:47.578176 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.578103    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kmgpj" podStartSLOduration=1.969199096 podStartE2EDuration="20.578089313s" totalImagesPullingTime="18.608890217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:47.577405317 +0000 UTC m=+22.623808262" watchObservedRunningTime="2026-09-17 18:29:47.578089313 +0000 UTC m=+22.624492214"
Sep 17 18:29:47.642197 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:47.642145    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pg2qm" podStartSLOduration=2.098416857 podStartE2EDuration="20.642132179s" totalImagesPullingTime="18.543715322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:47.641897505 +0000 UTC m=+22.688300408" watchObservedRunningTime="2026-09-17 18:29:47.642132179 +0000 UTC m=+22.688535080"
Sep 17 18:29:48.183324 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.183121    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:29:48.233435 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.233397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:48.233435 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.233419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:48.233652 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.233411    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:48.233652 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:48.233521    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:48.233652 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:48.233635    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:48.233788 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:48.233691    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:48.498854 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.498819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xgsr8" event={"ID":"299a3d68-caa2-4dd3-bd34-3f69bbceba8e","Type":"ContainerStarted","Data":"0188afffcdeead4c593742815004554b6319132b0eea738e6bdd9b1293793f75"}
Sep 17 18:29:48.501780 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.501754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq" event={"ID":"50caee2e-986c-47b4-9198-ae60bbd7bd90","Type":"ContainerStarted","Data":"af1b826a3f74316156186c6cb0c48b46c39668120f1f594c8bcfb3fcc6aca926"}
Sep 17 18:29:48.501897 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.501818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq" event={"ID":"50caee2e-986c-47b4-9198-ae60bbd7bd90","Type":"ContainerStarted","Data":"7ef17a90a6baab80d08a1b09a158e79a60194d5eb7e59fb4db4e9546d20fc305"}
Sep 17 18:29:48.503211 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.503188    2577 generic.go:356] "Generic (PLEG): container finished" podID="0833754b-b18a-43ba-bb8d-c5f1c4591a7a" containerID="6de0095b54ca5cb2a107d8d57061a45b5750a7f9595277250d3e1cc5ff801a70" exitCode=0
Sep 17 18:29:48.503277 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.503241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpp64" event={"ID":"0833754b-b18a-43ba-bb8d-c5f1c4591a7a","Type":"ContainerDied","Data":"6de0095b54ca5cb2a107d8d57061a45b5750a7f9595277250d3e1cc5ff801a70"}
Sep 17 18:29:48.505195 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.505176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r" event={"ID":"b122bb58-8686-4cad-a65d-6c4c8ae5650e","Type":"ContainerStarted","Data":"3927c0d98563b10ceea2a6c6fa9fb91e05b703d10b44bb71593f89eb1a129ab0"}
Sep 17 18:29:48.506962 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.506942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal" event={"ID":"ecc6e94cce9716a0525faaf0d6791fbd","Type":"ContainerStarted","Data":"3fd6216bbdc539fbabea469c6548ed43e3f2af019cdf30fd3c7003f05815c2a2"}
Sep 17 18:29:48.513111 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.513072    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xgsr8" podStartSLOduration=3.06772654 podStartE2EDuration="21.513059806s" totalImagesPullingTime="18.445333266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:48.512863301 +0000 UTC m=+23.559266202" watchObservedRunningTime="2026-09-17 18:29:48.513059806 +0000 UTC m=+23.559462708"
Sep 17 18:29:48.529563 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:48.529525    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-128.ec2.internal" podStartSLOduration=20.529514507000002 podStartE2EDuration="20.529514507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:48.52909427 +0000 UTC m=+23.575497172" watchObservedRunningTime="2026-09-17 18:29:48.529514507 +0000 UTC m=+23.575917410"
Sep 17 18:29:49.176995 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:49.176850    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:29:48.183320357Z","UUID":"fa239417-52ff-4558-b839-f9dd024f237d","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:29:49.178708 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:49.178687    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:29:49.178825 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:49.178720    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:29:49.512382 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:49.512332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r" event={"ID":"b122bb58-8686-4cad-a65d-6c4c8ae5650e","Type":"ContainerStarted","Data":"94c3e1d20a8e85dc54e76e0447c2d24281194607bbad36503fab861bb64f80e1"}
Sep 17 18:29:49.512810 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:49.512790    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 18:29:49.531122 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:49.531084    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltk5r" podStartSLOduration=2.528387377 podStartE2EDuration="22.531071116s" totalImagesPullingTime="20.002683739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:49.53056581 +0000 UTC m=+24.576968748" watchObservedRunningTime="2026-09-17 18:29:49.531071116 +0000 UTC m=+24.577474018"
Sep 17 18:29:50.232883 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:50.232850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:50.233062 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:50.232854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:50.233062 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:50.232973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:50.233062 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:50.232989    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:50.233206 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:50.233070    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:50.233206 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:50.233124    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:50.518346 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:50.518261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq" event={"ID":"50caee2e-986c-47b4-9198-ae60bbd7bd90","Type":"ContainerStarted","Data":"870217c2a98181a9b044f8624d0c5d4cf61363fb0972f12e3e5a4c1b63b750a9"}
Sep 17 18:29:52.233191 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:52.233157    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:52.233606 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:52.233216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:52.233606 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:52.233322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:52.233606 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:52.233323    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:52.233606 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:52.233404    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:52.233606 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:52.233513    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:53.528302 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:53.528065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq" event={"ID":"50caee2e-986c-47b4-9198-ae60bbd7bd90","Type":"ContainerStarted","Data":"3259e28f816ffade64fffdafa6cbff89f2ad68c670b64ec82a7b81854b653717"}
Sep 17 18:29:53.529028 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:53.528354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:53.529028 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:53.528404    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:53.530138 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:53.530114    2577 generic.go:356] "Generic (PLEG): container finished" podID="0833754b-b18a-43ba-bb8d-c5f1c4591a7a" containerID="5ceb76089c33082b7ef8e275834eeff4205909f1e3991bdec2f563387e7e970b" exitCode=0
Sep 17 18:29:53.530287 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:53.530162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpp64" event={"ID":"0833754b-b18a-43ba-bb8d-c5f1c4591a7a","Type":"ContainerDied","Data":"5ceb76089c33082b7ef8e275834eeff4205909f1e3991bdec2f563387e7e970b"}
Sep 17 18:29:53.544236 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:53.544214    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:53.558286 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:53.558227    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq" podStartSLOduration=7.899422943 podStartE2EDuration="26.558214929s" totalImagesPullingTime="18.658791986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:53.557632049 +0000 UTC m=+28.604034990" watchObservedRunningTime="2026-09-17 18:29:53.558214929 +0000 UTC m=+28.604617830"
Sep 17 18:29:54.233537 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.233512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:54.233700 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.233513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:54.233700 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.233642    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:54.233700 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:54.233641    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:54.233847 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:54.233765    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:54.233922 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:54.233854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:54.531827 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.531754    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cx897"]
Sep 17 18:29:54.532442 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.532408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j94pq"]
Sep 17 18:29:54.534727 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.534703    2577 generic.go:356] "Generic (PLEG): container finished" podID="0833754b-b18a-43ba-bb8d-c5f1c4591a7a" containerID="65c7dd7b0cdc50f2afecf312bcdcf9030f11aa2923e6d6a12bfadc5a46ca55cc" exitCode=0
Sep 17 18:29:54.534839 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.534799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpp64" event={"ID":"0833754b-b18a-43ba-bb8d-c5f1c4591a7a","Type":"ContainerDied","Data":"65c7dd7b0cdc50f2afecf312bcdcf9030f11aa2923e6d6a12bfadc5a46ca55cc"}
Sep 17 18:29:54.534925 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.534815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:54.534985 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:54.534969    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:54.535032 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.535023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:54.535115 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:54.535097    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:54.535674 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.535558    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:54.551101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.551081    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:29:54.552978 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.552960    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mqx5t"]
Sep 17 18:29:54.553054 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:54.553033    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:54.553112 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:54.553098    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:55.539563 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:55.539382    2577 generic.go:356] "Generic (PLEG): container finished" podID="0833754b-b18a-43ba-bb8d-c5f1c4591a7a" containerID="88bfb065e43285f912ebbffc6a84808179119e358309235d4ff64df0cbe9f96a" exitCode=0
Sep 17 18:29:55.539934 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:55.539465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpp64" event={"ID":"0833754b-b18a-43ba-bb8d-c5f1c4591a7a","Type":"ContainerDied","Data":"88bfb065e43285f912ebbffc6a84808179119e358309235d4ff64df0cbe9f96a"}
Sep 17 18:29:56.232830 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:56.232731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:56.232830 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:56.232763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:56.232830 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:56.232731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:56.233074 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:56.232856    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:56.233074 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:56.232949    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:56.233074 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:56.233053    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:57.472551 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:57.472510    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:57.473005 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:57.472639    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 18:29:57.473155 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:57.473136    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zn5q6"
Sep 17 18:29:58.233325 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:58.233293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:58.233455 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:58.233301    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:58.233455 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:58.233309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:29:58.233455 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:58.233431    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j94pq" podUID="ebc60a41-a81e-404d-978f-102cc369618b"
Sep 17 18:29:58.233560 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:58.233479    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqx5t" podUID="656e881f-1982-4ce2-a57c-299fb552772b"
Sep 17 18:29:58.233591 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:58.233555    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cx897" podUID="21c07112-ca6a-4083-9fad-f0b5b5afb0ba"
Sep 17 18:29:58.813935 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:58.813890    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pg2qm_a93c6cc9-d649-48f7-94fd-66df7247177c/dns-node-resolver/0.log"
Sep 17 18:29:59.753390 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:59.753365    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-128.ec2.internal" event="NodeReady"
Sep 17 18:29:59.753545 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:59.753476    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:29:59.784399 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:59.784377    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x7n4t_8e8f8317-5d67-48a6-81f8-4856de66a377/node-ca/0.log"
Sep 17 18:29:59.947335 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:59.947304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:29:59.947746 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:29:59.947349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:29:59.947746 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:59.947452    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:59.947746 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:59.947478    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:59.947746 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:59.947503    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs podName:656e881f-1982-4ce2-a57c-299fb552772b nodeName:}" failed. No retries permitted until 2026-09-17 18:30:31.947490316 +0000 UTC m=+66.993893195 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs") pod "network-metrics-daemon-mqx5t" (UID: "656e881f-1982-4ce2-a57c-299fb552772b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:59.947746 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:29:59.947540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret podName:ebc60a41-a81e-404d-978f-102cc369618b nodeName:}" failed. No retries permitted until 2026-09-17 18:30:31.94752307 +0000 UTC m=+66.993925962 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret") pod "global-pull-secret-syncer-j94pq" (UID: "ebc60a41-a81e-404d-978f-102cc369618b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:30:00.048554 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.048532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btshj\" (UniqueName: \"kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj\") pod \"network-check-target-cx897\" (UID: \"21c07112-ca6a-4083-9fad-f0b5b5afb0ba\") " pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:30:00.048706 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:30:00.048645    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:30:00.048706 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:30:00.048660    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:30:00.048706 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:30:00.048668    2577 projected.go:196] Error preparing data for projected volume kube-api-access-btshj for pod openshift-network-diagnostics/network-check-target-cx897: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:30:00.048800 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:30:00.048706    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj podName:21c07112-ca6a-4083-9fad-f0b5b5afb0ba nodeName:}" failed. No retries permitted until 2026-09-17 18:30:32.04869719 +0000 UTC m=+67.095100069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-btshj" (UniqueName: "kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj") pod "network-check-target-cx897" (UID: "21c07112-ca6a-4083-9fad-f0b5b5afb0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:30:00.232533 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.232511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:30:00.232671 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.232614    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:30:00.232671 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.232647    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:30:00.235613 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.235592    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:00.236773 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.236753    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ln4zj\""
Sep 17 18:30:00.236860 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.236783    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-crd74\""
Sep 17 18:30:00.236860 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.236811    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:00.237044 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.237030    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:30:00.237758 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.237741    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:30:00.555001 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.554967    2577 generic.go:356] "Generic (PLEG): container finished" podID="0833754b-b18a-43ba-bb8d-c5f1c4591a7a" containerID="90ba1627e4789160a4ee46fbaae7429c1359345926ab79040ddddc9bf8234fde" exitCode=0
Sep 17 18:30:00.555157 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:00.555026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpp64" event={"ID":"0833754b-b18a-43ba-bb8d-c5f1c4591a7a","Type":"ContainerDied","Data":"90ba1627e4789160a4ee46fbaae7429c1359345926ab79040ddddc9bf8234fde"}
Sep 17 18:30:01.560185 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:01.560005    2577 generic.go:356] "Generic (PLEG): container finished" podID="0833754b-b18a-43ba-bb8d-c5f1c4591a7a" containerID="59a0444e4f5e6cf3b35e22277bf02e0638698ef5e70c16e4b4d8b4d8e3007a2c" exitCode=0
Sep 17 18:30:01.560663 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:01.560081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpp64" event={"ID":"0833754b-b18a-43ba-bb8d-c5f1c4591a7a","Type":"ContainerDied","Data":"59a0444e4f5e6cf3b35e22277bf02e0638698ef5e70c16e4b4d8b4d8e3007a2c"}
Sep 17 18:30:02.565522 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:02.565489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpp64" event={"ID":"0833754b-b18a-43ba-bb8d-c5f1c4591a7a","Type":"ContainerStarted","Data":"18bb56afc4187c1623d7dff7f97db254994c832b7bff3a79f5dd75ef89031037"}
Sep 17 18:30:02.620473 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:02.620429    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xpp64" podStartSLOduration=6.954433047 podStartE2EDuration="35.62041137s" totalImagesPullingTime="28.665978323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:02.618645072 +0000 UTC m=+37.665047992" watchObservedRunningTime="2026-09-17 18:30:02.62041137 +0000 UTC m=+37.666814273"
Sep 17 18:30:22.608079 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.608051    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cff9d988d-5fl5c"]
Sep 17 18:30:22.612453 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.612432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.615666 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.615646    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:30:22.615760 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.615706    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:30:22.616705 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.616681    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5mhkt\""
Sep 17 18:30:22.621438 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.621414    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tlbhh"]
Sep 17 18:30:22.623236 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.623211    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:30:22.624629 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.624610    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:30:22.625594 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.625571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.627748 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.627729    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:30:22.628075 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.628045    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cff9d988d-5fl5c"]
Sep 17 18:30:22.628169 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.628125    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:30:22.628169 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.628135    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:30:22.628169 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.628135    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9cvxp\""
Sep 17 18:30:22.628442 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.628424    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:30:22.658083 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.658054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tlbhh"]
Sep 17 18:30:22.729603 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.729571    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rlt2p"]
Sep 17 18:30:22.733402 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.733383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rlt2p"
Sep 17 18:30:22.736071 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.736052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:30:22.736426 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.736409    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:30:22.736575 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.736563    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:30:22.747615 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.747598    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wggns\""
Sep 17 18:30:22.778628 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.778607    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tc2zd"]
Sep 17 18:30:22.784046 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-image-registry-private-configuration\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.784143 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-registry-certificates\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.784143 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9k8d\" (UniqueName: \"kubernetes.io/projected/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-kube-api-access-b9k8d\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.784143 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f00a5c9-c72a-490c-a33d-2dd121836a18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.784265 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-installation-pull-secrets\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.784265 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f00a5c9-c72a-490c-a33d-2dd121836a18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.784265 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmgm\" (UniqueName: \"kubernetes.io/projected/6f00a5c9-c72a-490c-a33d-2dd121836a18-kube-api-access-hdmgm\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.784265 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-registry-tls\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.784436 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f00a5c9-c72a-490c-a33d-2dd121836a18-data-volume\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.784436 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-trusted-ca\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.784436 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-bound-sa-token\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.784436 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6f00a5c9-c72a-490c-a33d-2dd121836a18-crio-socket\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.784584 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.784435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-ca-trust-extracted\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.798626 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.798603    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rlt2p"]
Sep 17 18:30:22.798748 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.798731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.799009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.798967    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tc2zd"]
Sep 17 18:30:22.805010 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.804991    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:30:22.806501 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.806479    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:30:22.815613 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.815596    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6cjgz\""
Sep 17 18:30:22.884726 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.884667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rz2r\" (UniqueName: \"kubernetes.io/projected/be5fcd92-12b9-43df-9e14-a0694aff9fe7-kube-api-access-7rz2r\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.884726 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.884702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-registry-tls\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.884726 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.884720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be5fcd92-12b9-43df-9e14-a0694aff9fe7-metrics-tls\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.884884 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.884735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be5fcd92-12b9-43df-9e14-a0694aff9fe7-tmp-dir\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.884884 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.884786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f00a5c9-c72a-490c-a33d-2dd121836a18-data-volume\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.884884 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.884814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-trusted-ca\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.884884 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.884831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-bound-sa-token\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.885086 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.884885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0885b1df-0695-4589-8ae5-387f4a25c048-cert\") pod \"ingress-canary-rlt2p\" (UID: \"0885b1df-0695-4589-8ae5-387f4a25c048\") " pod="openshift-ingress-canary/ingress-canary-rlt2p"
Sep 17 18:30:22.885086 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.884955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6f00a5c9-c72a-490c-a33d-2dd121836a18-crio-socket\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.885086 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.884981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-ca-trust-extracted\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.885086 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-image-registry-private-configuration\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.885086 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-registry-certificates\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.885086 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6f00a5c9-c72a-490c-a33d-2dd121836a18-crio-socket\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.885086 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5srhq\" (UniqueName: \"kubernetes.io/projected/0885b1df-0695-4589-8ae5-387f4a25c048-kube-api-access-5srhq\") pod \"ingress-canary-rlt2p\" (UID: \"0885b1df-0695-4589-8ae5-387f4a25c048\") " pod="openshift-ingress-canary/ingress-canary-rlt2p"
Sep 17 18:30:22.885378 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9k8d\" (UniqueName: \"kubernetes.io/projected/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-kube-api-access-b9k8d\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.885378 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f00a5c9-c72a-490c-a33d-2dd121836a18-data-volume\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.885378 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f00a5c9-c72a-490c-a33d-2dd121836a18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.885378 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-installation-pull-secrets\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.885378 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f00a5c9-c72a-490c-a33d-2dd121836a18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.885378 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmgm\" (UniqueName: \"kubernetes.io/projected/6f00a5c9-c72a-490c-a33d-2dd121836a18-kube-api-access-hdmgm\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.885378 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be5fcd92-12b9-43df-9e14-a0694aff9fe7-config-volume\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.885700 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-ca-trust-extracted\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.885820 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f00a5c9-c72a-490c-a33d-2dd121836a18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.885889 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885835    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-registry-certificates\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.885983 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.885965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-trusted-ca\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.889819 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.889802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f00a5c9-c72a-490c-a33d-2dd121836a18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.889894 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.889831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-image-registry-private-configuration\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.889957 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.889937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-registry-tls\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.889992 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.889956    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-installation-pull-secrets\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.898007 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.897988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-bound-sa-token\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.898498 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.898479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9k8d\" (UniqueName: \"kubernetes.io/projected/05daf5f5-08e0-4f7c-af04-eeed64c62fb5-kube-api-access-b9k8d\") pod \"image-registry-6cff9d988d-5fl5c\" (UID: \"05daf5f5-08e0-4f7c-af04-eeed64c62fb5\") " pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.898682 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.898662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmgm\" (UniqueName: \"kubernetes.io/projected/6f00a5c9-c72a-490c-a33d-2dd121836a18-kube-api-access-hdmgm\") pod \"insights-runtime-extractor-tlbhh\" (UID: \"6f00a5c9-c72a-490c-a33d-2dd121836a18\") " pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.922118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.922088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:22.934227 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.934211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tlbhh"
Sep 17 18:30:22.986198 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.986159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be5fcd92-12b9-43df-9e14-a0694aff9fe7-config-volume\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.986365 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.986212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rz2r\" (UniqueName: \"kubernetes.io/projected/be5fcd92-12b9-43df-9e14-a0694aff9fe7-kube-api-access-7rz2r\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.986365 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.986250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be5fcd92-12b9-43df-9e14-a0694aff9fe7-metrics-tls\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.986365 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.986276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be5fcd92-12b9-43df-9e14-a0694aff9fe7-tmp-dir\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.986365 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.986311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0885b1df-0695-4589-8ae5-387f4a25c048-cert\") pod \"ingress-canary-rlt2p\" (UID: \"0885b1df-0695-4589-8ae5-387f4a25c048\") " pod="openshift-ingress-canary/ingress-canary-rlt2p"
Sep 17 18:30:22.986365 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.986349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srhq\" (UniqueName: \"kubernetes.io/projected/0885b1df-0695-4589-8ae5-387f4a25c048-kube-api-access-5srhq\") pod \"ingress-canary-rlt2p\" (UID: \"0885b1df-0695-4589-8ae5-387f4a25c048\") " pod="openshift-ingress-canary/ingress-canary-rlt2p"
Sep 17 18:30:22.987176 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.986648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be5fcd92-12b9-43df-9e14-a0694aff9fe7-tmp-dir\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.987176 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.986677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be5fcd92-12b9-43df-9e14-a0694aff9fe7-config-volume\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.988738 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.988718    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be5fcd92-12b9-43df-9e14-a0694aff9fe7-metrics-tls\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.988842 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.988722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0885b1df-0695-4589-8ae5-387f4a25c048-cert\") pod \"ingress-canary-rlt2p\" (UID: \"0885b1df-0695-4589-8ae5-387f4a25c048\") " pod="openshift-ingress-canary/ingress-canary-rlt2p"
Sep 17 18:30:22.997844 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.997816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rz2r\" (UniqueName: \"kubernetes.io/projected/be5fcd92-12b9-43df-9e14-a0694aff9fe7-kube-api-access-7rz2r\") pod \"dns-default-tc2zd\" (UID: \"be5fcd92-12b9-43df-9e14-a0694aff9fe7\") " pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:22.998262 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:22.998246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5srhq\" (UniqueName: \"kubernetes.io/projected/0885b1df-0695-4589-8ae5-387f4a25c048-kube-api-access-5srhq\") pod \"ingress-canary-rlt2p\" (UID: \"0885b1df-0695-4589-8ae5-387f4a25c048\") " pod="openshift-ingress-canary/ingress-canary-rlt2p"
Sep 17 18:30:23.041536 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.041504    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rlt2p"
Sep 17 18:30:23.107486 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.107188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:23.271974 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.271942    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tlbhh"]
Sep 17 18:30:23.275263 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.275239    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cff9d988d-5fl5c"]
Sep 17 18:30:23.275428 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:23.275409    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f00a5c9_c72a_490c_a33d_2dd121836a18.slice/crio-4b8787384b10c66dae3767ed541b1fddeb66e4c4169b0a35426e7c2bce326046 WatchSource:0}: Error finding container 4b8787384b10c66dae3767ed541b1fddeb66e4c4169b0a35426e7c2bce326046: Status 404 returned error can't find the container with id 4b8787384b10c66dae3767ed541b1fddeb66e4c4169b0a35426e7c2bce326046
Sep 17 18:30:23.278493 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:23.278438    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05daf5f5_08e0_4f7c_af04_eeed64c62fb5.slice/crio-36eada5f7d09996ff4449fca1a8f1ec0ce62ab6ba648babb07c2c2436c7ad806 WatchSource:0}: Error finding container 36eada5f7d09996ff4449fca1a8f1ec0ce62ab6ba648babb07c2c2436c7ad806: Status 404 returned error can't find the container with id 36eada5f7d09996ff4449fca1a8f1ec0ce62ab6ba648babb07c2c2436c7ad806
Sep 17 18:30:23.329384 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.329182    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rlt2p"]
Sep 17 18:30:23.332699 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:23.332670    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0885b1df_0695_4589_8ae5_387f4a25c048.slice/crio-d877d3918ea81ef5ace4bfafc5a4dbcf48d94c4922364332f96e95cd544880d4 WatchSource:0}: Error finding container d877d3918ea81ef5ace4bfafc5a4dbcf48d94c4922364332f96e95cd544880d4: Status 404 returned error can't find the container with id d877d3918ea81ef5ace4bfafc5a4dbcf48d94c4922364332f96e95cd544880d4
Sep 17 18:30:23.392815 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.391011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tc2zd"]
Sep 17 18:30:23.395247 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:23.395147    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5fcd92_12b9_43df_9e14_a0694aff9fe7.slice/crio-c3e23712df4454a12c3a2fc7020d1e1474cff6047c17fdaa1a19962a539f90d2 WatchSource:0}: Error finding container c3e23712df4454a12c3a2fc7020d1e1474cff6047c17fdaa1a19962a539f90d2: Status 404 returned error can't find the container with id c3e23712df4454a12c3a2fc7020d1e1474cff6047c17fdaa1a19962a539f90d2
Sep 17 18:30:23.614847 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.614814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tc2zd" event={"ID":"be5fcd92-12b9-43df-9e14-a0694aff9fe7","Type":"ContainerStarted","Data":"c3e23712df4454a12c3a2fc7020d1e1474cff6047c17fdaa1a19962a539f90d2"}
Sep 17 18:30:23.616000 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.615969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rlt2p" event={"ID":"0885b1df-0695-4589-8ae5-387f4a25c048","Type":"ContainerStarted","Data":"d877d3918ea81ef5ace4bfafc5a4dbcf48d94c4922364332f96e95cd544880d4"}
Sep 17 18:30:23.617454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.617427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c" event={"ID":"05daf5f5-08e0-4f7c-af04-eeed64c62fb5","Type":"ContainerStarted","Data":"473209ce3f9b37fdb6a0a27dc2da11e04914c2f5cdc7fc284b2667ac695a29ec"}
Sep 17 18:30:23.617548 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.617476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c" event={"ID":"05daf5f5-08e0-4f7c-af04-eeed64c62fb5","Type":"ContainerStarted","Data":"36eada5f7d09996ff4449fca1a8f1ec0ce62ab6ba648babb07c2c2436c7ad806"}
Sep 17 18:30:23.617602 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.617547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:23.618916 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.618886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlbhh" event={"ID":"6f00a5c9-c72a-490c-a33d-2dd121836a18","Type":"ContainerStarted","Data":"3a44d40447b810b71210bf61db29d6c3a6a022e46ab26f57b1b2aedf215280c5"}
Sep 17 18:30:23.619011 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.618952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlbhh" event={"ID":"6f00a5c9-c72a-490c-a33d-2dd121836a18","Type":"ContainerStarted","Data":"4b8787384b10c66dae3767ed541b1fddeb66e4c4169b0a35426e7c2bce326046"}
Sep 17 18:30:23.637405 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:23.637361    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c" podStartSLOduration=1.637348501 podStartE2EDuration="1.637348501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:23.636403261 +0000 UTC m=+58.682806191" watchObservedRunningTime="2026-09-17 18:30:23.637348501 +0000 UTC m=+58.683751442"
Sep 17 18:30:24.624341 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:24.624260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlbhh" event={"ID":"6f00a5c9-c72a-490c-a33d-2dd121836a18","Type":"ContainerStarted","Data":"1f88e4034a3bdbb90b0b229ba5c2cb7ab9dfb44955c476280267b4c6614c8d46"}
Sep 17 18:30:26.035393 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.035354    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"]
Sep 17 18:30:26.038029 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.038004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.041788 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.041754    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:30:26.041886 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.041754    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:30:26.043033 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.043010    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 18:30:26.043156 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.043137    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 18:30:26.043230 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.043158    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:30:26.043230 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.043202    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pcggs\""
Sep 17 18:30:26.058889 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.058868    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"]
Sep 17 18:30:26.213129 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.213072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.213241 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.213192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dkpc\" (UniqueName: \"kubernetes.io/projected/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-kube-api-access-6dkpc\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.213296 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.213256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.213368 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.213351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.314726 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.314273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.314726 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.314361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.314726 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.314397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dkpc\" (UniqueName: \"kubernetes.io/projected/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-kube-api-access-6dkpc\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.314726 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.314444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.315404 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.315351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.318894 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.318862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.320008 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.318862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.324250 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.324207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dkpc\" (UniqueName: \"kubernetes.io/projected/c8414deb-4ab1-4aae-8f38-a48e97d6dfd5-kube-api-access-6dkpc\") pod \"prometheus-operator-7bd5d56d6-qntw7\" (UID: \"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.348201 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.348179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"
Sep 17 18:30:26.552774 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.552698    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5bqq"
Sep 17 18:30:26.629804 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.629759    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7"]
Sep 17 18:30:26.633272 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.633245    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlbhh" event={"ID":"6f00a5c9-c72a-490c-a33d-2dd121836a18","Type":"ContainerStarted","Data":"9bd6bc9e8cd106b040ee0388886d78d19bc6a80ee8e67a649c9887cd7c3061b8"}
Sep 17 18:30:26.633731 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:26.633704    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8414deb_4ab1_4aae_8f38_a48e97d6dfd5.slice/crio-c200af9d4e1aef4d650385f313165ab7d576323072ad690a9b687fae68fe07b6 WatchSource:0}: Error finding container c200af9d4e1aef4d650385f313165ab7d576323072ad690a9b687fae68fe07b6: Status 404 returned error can't find the container with id c200af9d4e1aef4d650385f313165ab7d576323072ad690a9b687fae68fe07b6
Sep 17 18:30:26.635267 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.635241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tc2zd" event={"ID":"be5fcd92-12b9-43df-9e14-a0694aff9fe7","Type":"ContainerStarted","Data":"4e22229255272050cfba72df475e173362fcda96fe2d910b31251f1233f1a84a"}
Sep 17 18:30:26.635378 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.635292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tc2zd" event={"ID":"be5fcd92-12b9-43df-9e14-a0694aff9fe7","Type":"ContainerStarted","Data":"328e70eff35611396277226963e9cc01167ad927a28e83d9392e2dbdccf7bc0c"}
Sep 17 18:30:26.635378 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.635321    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:26.636839 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.636811    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rlt2p" event={"ID":"0885b1df-0695-4589-8ae5-387f4a25c048","Type":"ContainerStarted","Data":"5b9f5eb50c042b8e05f212efb63ac4ac03d42bfe8ea8a66888549f2e5499662a"}
Sep 17 18:30:26.659470 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.659431    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tlbhh" podStartSLOduration=1.987154473 podStartE2EDuration="4.659419847s" totalImagesPullingTime="2.672265374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:26.658734282 +0000 UTC m=+61.705137213" watchObservedRunningTime="2026-09-17 18:30:26.659419847 +0000 UTC m=+61.705822748"
Sep 17 18:30:26.687523 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.687485    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tc2zd" podStartSLOduration=1.9091180300000001 podStartE2EDuration="4.687471515s" totalImagesPullingTime="2.778353485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:26.686950563 +0000 UTC m=+61.733353473" watchObservedRunningTime="2026-09-17 18:30:26.687471515 +0000 UTC m=+61.733874415"
Sep 17 18:30:26.716398 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:26.716365    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rlt2p" podStartSLOduration=1.867814027 podStartE2EDuration="4.716354083s" totalImagesPullingTime="2.848540056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:26.714591655 +0000 UTC m=+61.760994578" watchObservedRunningTime="2026-09-17 18:30:26.716354083 +0000 UTC m=+61.762756984"
Sep 17 18:30:27.641488 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:27.641412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7" event={"ID":"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5","Type":"ContainerStarted","Data":"c200af9d4e1aef4d650385f313165ab7d576323072ad690a9b687fae68fe07b6"}
Sep 17 18:30:28.359431 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.359383    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6497ddd774-twfmn"]
Sep 17 18:30:28.361314 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.361276    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.367437 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.367404    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 18:30:28.367567 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.367457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 18:30:28.368735 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.368720    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 18:30:28.368821 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.368732    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5gxlh\""
Sep 17 18:30:28.369199 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.369181    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 18:30:28.369258 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.369219    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 18:30:28.369258 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.369172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 18:30:28.369523 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.369511    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 18:30:28.388427 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.388401    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6497ddd774-twfmn"]
Sep 17 18:30:28.435972 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.433531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vv7q\" (UniqueName: \"kubernetes.io/projected/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-kube-api-access-2vv7q\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.435972 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.433590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-serving-cert\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.435972 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.433664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-oauth-serving-cert\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.435972 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.433705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-oauth-config\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.435972 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.433751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-service-ca\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.435972 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.433782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-config\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.534537 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.534499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-oauth-serving-cert\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.534537 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.534552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-oauth-config\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.534836 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.534585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-service-ca\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.534836 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.534603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-config\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.534836 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.534654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vv7q\" (UniqueName: \"kubernetes.io/projected/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-kube-api-access-2vv7q\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.534836 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.534679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-serving-cert\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.535349 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.535326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-config\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.535452 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.535390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-oauth-serving-cert\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.535452 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.535440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-service-ca\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.537128 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.537107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-serving-cert\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.537233 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.537212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-oauth-config\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.543522 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.543503    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vv7q\" (UniqueName: \"kubernetes.io/projected/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-kube-api-access-2vv7q\") pod \"console-6497ddd774-twfmn\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") " pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.645961 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.645878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7" event={"ID":"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5","Type":"ContainerStarted","Data":"71b859954e79653d660bd6391a0e3c70b9ee8f59ae3851451f15eb2f3e9f4836"}
Sep 17 18:30:28.646311 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.645963    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7" event={"ID":"c8414deb-4ab1-4aae-8f38-a48e97d6dfd5","Type":"ContainerStarted","Data":"3482398ef3c9f37085a2e4e3f6856e0035a3789e93108699c4c6047733945d0e"}
Sep 17 18:30:28.670282 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.670260    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:28.676878 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.676832    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-qntw7" podStartSLOduration=1.199993123 podStartE2EDuration="2.676819122s" totalImagesPullingTime="1.476825999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:28.67619876 +0000 UTC m=+63.722601660" watchObservedRunningTime="2026-09-17 18:30:28.676819122 +0000 UTC m=+63.723222063"
Sep 17 18:30:28.959412 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:28.959371    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6497ddd774-twfmn"]
Sep 17 18:30:28.962172 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:28.962127    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e0d2c8_8c39_4b9b_9ae4_4512e86406b3.slice/crio-38fa5706c29771219e40f164747f59f9100d4a50a60a524fca0a754afb34124e WatchSource:0}: Error finding container 38fa5706c29771219e40f164747f59f9100d4a50a60a524fca0a754afb34124e: Status 404 returned error can't find the container with id 38fa5706c29771219e40f164747f59f9100d4a50a60a524fca0a754afb34124e
Sep 17 18:30:29.651393 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:29.651338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6497ddd774-twfmn" event={"ID":"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3","Type":"ContainerStarted","Data":"38fa5706c29771219e40f164747f59f9100d4a50a60a524fca0a754afb34124e"}
Sep 17 18:30:30.527523 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.527487    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"]
Sep 17 18:30:30.558349 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.558192    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-585cx"]
Sep 17 18:30:30.558349 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.558312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.568792 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.568608    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 18:30:30.570485 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.570292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-r9kmq\""
Sep 17 18:30:30.570609 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.570496    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 18:30:30.592495 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.592468    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7rssq"]
Sep 17 18:30:30.592691 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.592669    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.595640 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.595622    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 18:30:30.596130 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.595952    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bdtl8\""
Sep 17 18:30:30.596130 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.596054    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 18:30:30.596130 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.596062    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 18:30:30.624669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.624643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"]
Sep 17 18:30:30.624792 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.624690    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-585cx"]
Sep 17 18:30:30.624792 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.624776    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.633822 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.632860    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:30:30.633822 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.633561    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:30:30.633822 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.633662    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6gdgr\""
Sep 17 18:30:30.633822 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.633759    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:30:30.650681 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.650654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-tls\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.650804 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.650696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/998c753e-fa09-4b9e-b81b-7deb8aaabfec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.650804 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.650732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72dd61e0-804b-4b85-abd1-eafbb220388f-sys\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.650950 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.650822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf445\" (UniqueName: \"kubernetes.io/projected/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-api-access-mf445\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.650950 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.650878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm4lg\" (UniqueName: \"kubernetes.io/projected/72dd61e0-804b-4b85-abd1-eafbb220388f-kube-api-access-vm4lg\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.651078 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.650953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-textfile\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.651078 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.650983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3ba0a29-5045-4cf7-9008-998151191fb7-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.651078 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.651078 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72dd61e0-804b-4b85-abd1-eafbb220388f-metrics-client-ca\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.651287 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72dd61e0-804b-4b85-abd1-eafbb220388f-root\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.651287 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/998c753e-fa09-4b9e-b81b-7deb8aaabfec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.651287 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/998c753e-fa09-4b9e-b81b-7deb8aaabfec-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.651287 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndpc\" (UniqueName: \"kubernetes.io/projected/998c753e-fa09-4b9e-b81b-7deb8aaabfec-kube-api-access-tndpc\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.651287 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.651287 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b3ba0a29-5045-4cf7-9008-998151191fb7-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.651887 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.651887 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.651887 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.651887 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.651391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-wtmp\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.752809 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.752768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.752974 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.752833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72dd61e0-804b-4b85-abd1-eafbb220388f-metrics-client-ca\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.752974 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.752861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72dd61e0-804b-4b85-abd1-eafbb220388f-root\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.752974 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.752885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/998c753e-fa09-4b9e-b81b-7deb8aaabfec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.752974 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.752931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/998c753e-fa09-4b9e-b81b-7deb8aaabfec-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.752974 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.752962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndpc\" (UniqueName: \"kubernetes.io/projected/998c753e-fa09-4b9e-b81b-7deb8aaabfec-kube-api-access-tndpc\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.753241 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.752985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.753241 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b3ba0a29-5045-4cf7-9008-998151191fb7-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.753241 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72dd61e0-804b-4b85-abd1-eafbb220388f-root\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.753241 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.753241 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.753241 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.753510 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-wtmp\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.753510 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-tls\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.753510 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/998c753e-fa09-4b9e-b81b-7deb8aaabfec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.753510 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72dd61e0-804b-4b85-abd1-eafbb220388f-sys\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.753510 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf445\" (UniqueName: \"kubernetes.io/projected/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-api-access-mf445\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.753510 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b3ba0a29-5045-4cf7-9008-998151191fb7-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.753510 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm4lg\" (UniqueName: \"kubernetes.io/projected/72dd61e0-804b-4b85-abd1-eafbb220388f-kube-api-access-vm4lg\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.753510 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-textfile\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.753510 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3ba0a29-5045-4cf7-9008-998151191fb7-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.753937 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72dd61e0-804b-4b85-abd1-eafbb220388f-metrics-client-ca\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.753937 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.753918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72dd61e0-804b-4b85-abd1-eafbb220388f-sys\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.754053 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.754010    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/998c753e-fa09-4b9e-b81b-7deb8aaabfec-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.754714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.754219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3ba0a29-5045-4cf7-9008-998151191fb7-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.754714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.754356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-textfile\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.754714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.754396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-wtmp\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.754714 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:30:30.754452    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 18:30:30.754714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.754507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.754714 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:30:30.754522    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-tls podName:72dd61e0-804b-4b85-abd1-eafbb220388f nodeName:}" failed. No retries permitted until 2026-09-17 18:30:31.254504205 +0000 UTC m=+66.300907099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-tls") pod "node-exporter-7rssq" (UID: "72dd61e0-804b-4b85-abd1-eafbb220388f") : secret "node-exporter-tls" not found
Sep 17 18:30:30.754714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.754569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.756260 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.756235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.756931 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.756877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/998c753e-fa09-4b9e-b81b-7deb8aaabfec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.757044 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.756964    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/998c753e-fa09-4b9e-b81b-7deb8aaabfec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.758165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.758121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.758565 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.758535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.767108 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.767086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm4lg\" (UniqueName: \"kubernetes.io/projected/72dd61e0-804b-4b85-abd1-eafbb220388f-kube-api-access-vm4lg\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:30.767675 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.767632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf445\" (UniqueName: \"kubernetes.io/projected/b3ba0a29-5045-4cf7-9008-998151191fb7-kube-api-access-mf445\") pod \"kube-state-metrics-84c885658c-585cx\" (UID: \"b3ba0a29-5045-4cf7-9008-998151191fb7\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:30.767783 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.767697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndpc\" (UniqueName: \"kubernetes.io/projected/998c753e-fa09-4b9e-b81b-7deb8aaabfec-kube-api-access-tndpc\") pod \"openshift-state-metrics-86c5bc79d6-6wm5x\" (UID: \"998c753e-fa09-4b9e-b81b-7deb8aaabfec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.871594 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.871524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"
Sep 17 18:30:30.904858 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:30.904825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx"
Sep 17 18:30:31.257939 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.257884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-tls\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:31.260623 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.260596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72dd61e0-804b-4b85-abd1-eafbb220388f-node-exporter-tls\") pod \"node-exporter-7rssq\" (UID: \"72dd61e0-804b-4b85-abd1-eafbb220388f\") " pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:31.454471 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.454431    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:30:31.478470 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.478439    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:30:31.478658 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.478621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.486626 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.486605    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 18:30:31.486751 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.486607    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 18:30:31.486751 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.486607    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 18:30:31.487063 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.487045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 18:30:31.487396 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.487380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 18:30:31.487632 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.487617    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zlj98\""
Sep 17 18:30:31.487707 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.487620    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 18:30:31.487788 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.487773    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 18:30:31.489129 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.489116    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 18:30:31.489989 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.489975    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 18:30:31.536439 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.536383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7rssq"
Sep 17 18:30:31.561402 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561459 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rxhj\" (UniqueName: \"kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-kube-api-access-2rxhj\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561516 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561572 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561647 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561680 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561710 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-web-config\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561744 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-out\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561775 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561805 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561834 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561864 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.561864 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.561851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663174 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rxhj\" (UniqueName: \"kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-kube-api-access-2rxhj\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663627 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663627 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663627 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663627 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663627 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-web-config\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663938 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-out\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663938 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663938 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663938 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663938 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663938 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.663787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.663938 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:30:31.663849    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 18:30:31.663938 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:30:31.663932    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-main-tls podName:29e59426-8fd1-4741-a6c2-dbd7300641a2 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:32.163891104 +0000 UTC m=+67.210293985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2") : secret "alertmanager-main-tls" not found
Sep 17 18:30:31.664367 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.664343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.664435 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.664419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.664724 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.664702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.664800 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.664702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.666386 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.666362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-web-config\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.666646 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.666626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.667136 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.667084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.667826 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.667734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.667826 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.667799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-out\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.668229 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.668189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.668336 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.668316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.668419 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.668320    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.678369 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.678347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rxhj\" (UniqueName: \"kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-kube-api-access-2rxhj\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:31.966869 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.966821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:30:31.967064 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.966885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:30:31.969965 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.969943    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:31.970088 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.969944    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:31.979972 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.979950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebc60a41-a81e-404d-978f-102cc369618b-original-pull-secret\") pod \"global-pull-secret-syncer-j94pq\" (UID: \"ebc60a41-a81e-404d-978f-102cc369618b\") " pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:30:31.980056 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:31.980012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656e881f-1982-4ce2-a57c-299fb552772b-metrics-certs\") pod \"network-metrics-daemon-mqx5t\" (UID: \"656e881f-1982-4ce2-a57c-299fb552772b\") " pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:30:32.045851 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.045827    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-crd74\""
Sep 17 18:30:32.054167 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.054121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqx5t"
Sep 17 18:30:32.063546 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.063521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j94pq"
Sep 17 18:30:32.067938 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.067712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btshj\" (UniqueName: \"kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj\") pod \"network-check-target-cx897\" (UID: \"21c07112-ca6a-4083-9fad-f0b5b5afb0ba\") " pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:30:32.070537 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.070518    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:30:32.081222 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.081204    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:30:32.092299 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.092279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btshj\" (UniqueName: \"kubernetes.io/projected/21c07112-ca6a-4083-9fad-f0b5b5afb0ba-kube-api-access-btshj\") pod \"network-check-target-cx897\" (UID: \"21c07112-ca6a-4083-9fad-f0b5b5afb0ba\") " pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:30:32.168205 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.168175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:32.170770 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.170719    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:32.352785 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.352700    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ln4zj\""
Sep 17 18:30:32.360893 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.360868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:30:32.387667 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:32.387639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:33.282694 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:33.282663    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72dd61e0_804b_4b85_abd1_eafbb220388f.slice/crio-72a01be43fce911a2740dc826c5691eee5adad1234fb5caa514f9089ec59be02 WatchSource:0}: Error finding container 72a01be43fce911a2740dc826c5691eee5adad1234fb5caa514f9089ec59be02: Status 404 returned error can't find the container with id 72a01be43fce911a2740dc826c5691eee5adad1234fb5caa514f9089ec59be02
Sep 17 18:30:33.479170 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.479133    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"]
Sep 17 18:30:33.482269 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.482248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.485089 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.485064    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 18:30:33.485796 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.485654    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e354jb8662mbg\""
Sep 17 18:30:33.485796 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.485668    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 18:30:33.485796 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.485691    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 18:30:33.485796 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.485716    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 18:30:33.486170 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.486137    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-sp9q8\""
Sep 17 18:30:33.486295 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.486280    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 18:30:33.501649 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.501601    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"]
Sep 17 18:30:33.520816 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.520794    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-585cx"]
Sep 17 18:30:33.524378 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:33.524359    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ba0a29_5045_4cf7_9008_998151191fb7.slice/crio-7522e9efb7a787fe8a567ffd62d9c5415cfa059289de4fe9652bea62deb1debf WatchSource:0}: Error finding container 7522e9efb7a787fe8a567ffd62d9c5415cfa059289de4fe9652bea62deb1debf: Status 404 returned error can't find the container with id 7522e9efb7a787fe8a567ffd62d9c5415cfa059289de4fe9652bea62deb1debf
Sep 17 18:30:33.574084 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.573954    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j94pq"]
Sep 17 18:30:33.576655 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:33.576634    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc60a41_a81e_404d_978f_102cc369618b.slice/crio-6a11ec9834ef6f7caca214bb2eca0565970cab8971f5bebf8051d3ed7d975b90 WatchSource:0}: Error finding container 6a11ec9834ef6f7caca214bb2eca0565970cab8971f5bebf8051d3ed7d975b90: Status 404 returned error can't find the container with id 6a11ec9834ef6f7caca214bb2eca0565970cab8971f5bebf8051d3ed7d975b90
Sep 17 18:30:33.577247 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.577226    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x"]
Sep 17 18:30:33.579310 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:33.579291    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998c753e_fa09_4b9e_b81b_7deb8aaabfec.slice/crio-d8ac14d260ddcdd76264a0b76993cf7ebb20f65face758dbe79b85567e5c5436 WatchSource:0}: Error finding container d8ac14d260ddcdd76264a0b76993cf7ebb20f65face758dbe79b85567e5c5436: Status 404 returned error can't find the container with id d8ac14d260ddcdd76264a0b76993cf7ebb20f65face758dbe79b85567e5c5436
Sep 17 18:30:33.581672 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.581649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60025e0d-4f71-4be8-bbfa-8665f8cba082-metrics-client-ca\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.581778 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.581716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-grpc-tls\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.581841 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.581785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhdv4\" (UniqueName: \"kubernetes.io/projected/60025e0d-4f71-4be8-bbfa-8665f8cba082-kube-api-access-dhdv4\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.581841 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.581820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-tls\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.581955 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.581857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.581955 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.581920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.582207 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.582132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.582286 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.582225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.586966 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.586928    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mqx5t"]
Sep 17 18:30:33.589610 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:33.589593    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656e881f_1982_4ce2_a57c_299fb552772b.slice/crio-d77ce1be2c6002259906188e8b3f3da972021c801408041684f175bb700c2ee0 WatchSource:0}: Error finding container d77ce1be2c6002259906188e8b3f3da972021c801408041684f175bb700c2ee0: Status 404 returned error can't find the container with id d77ce1be2c6002259906188e8b3f3da972021c801408041684f175bb700c2ee0
Sep 17 18:30:33.594534 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.594515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cx897"]
Sep 17 18:30:33.596538 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:33.596513    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c07112_ca6a_4083_9fad_f0b5b5afb0ba.slice/crio-a2099407c22b33d70e787f424eb1706d0135a285e98e806a20401eb6d76ee0c0 WatchSource:0}: Error finding container a2099407c22b33d70e787f424eb1706d0135a285e98e806a20401eb6d76ee0c0: Status 404 returned error can't find the container with id a2099407c22b33d70e787f424eb1706d0135a285e98e806a20401eb6d76ee0c0
Sep 17 18:30:33.610434 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.610389    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:30:33.612414 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:33.612388    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e59426_8fd1_4741_a6c2_dbd7300641a2.slice/crio-90b3164bf55798464637a3f9d418128eedb0795dba036618c702006ec05aed92 WatchSource:0}: Error finding container 90b3164bf55798464637a3f9d418128eedb0795dba036618c702006ec05aed92: Status 404 returned error can't find the container with id 90b3164bf55798464637a3f9d418128eedb0795dba036618c702006ec05aed92
Sep 17 18:30:33.665837 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.665809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx" event={"ID":"b3ba0a29-5045-4cf7-9008-998151191fb7","Type":"ContainerStarted","Data":"7522e9efb7a787fe8a567ffd62d9c5415cfa059289de4fe9652bea62deb1debf"}
Sep 17 18:30:33.667575 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.667548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6497ddd774-twfmn" event={"ID":"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3","Type":"ContainerStarted","Data":"da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4"}
Sep 17 18:30:33.668959 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.668930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerStarted","Data":"90b3164bf55798464637a3f9d418128eedb0795dba036618c702006ec05aed92"}
Sep 17 18:30:33.670623 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.670595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j94pq" event={"ID":"ebc60a41-a81e-404d-978f-102cc369618b","Type":"ContainerStarted","Data":"6a11ec9834ef6f7caca214bb2eca0565970cab8971f5bebf8051d3ed7d975b90"}
Sep 17 18:30:33.673025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.673000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cx897" event={"ID":"21c07112-ca6a-4083-9fad-f0b5b5afb0ba","Type":"ContainerStarted","Data":"a2099407c22b33d70e787f424eb1706d0135a285e98e806a20401eb6d76ee0c0"}
Sep 17 18:30:33.674340 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.674315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rssq" event={"ID":"72dd61e0-804b-4b85-abd1-eafbb220388f","Type":"ContainerStarted","Data":"72a01be43fce911a2740dc826c5691eee5adad1234fb5caa514f9089ec59be02"}
Sep 17 18:30:33.675614 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.675588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mqx5t" event={"ID":"656e881f-1982-4ce2-a57c-299fb552772b","Type":"ContainerStarted","Data":"d77ce1be2c6002259906188e8b3f3da972021c801408041684f175bb700c2ee0"}
Sep 17 18:30:33.676947 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.676928    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x" event={"ID":"998c753e-fa09-4b9e-b81b-7deb8aaabfec","Type":"ContainerStarted","Data":"300d1dabfc42a276fb152031a6cfc4d952351f3fbe25ecc576dd11033e87acc7"}
Sep 17 18:30:33.677026 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.676968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x" event={"ID":"998c753e-fa09-4b9e-b81b-7deb8aaabfec","Type":"ContainerStarted","Data":"d8ac14d260ddcdd76264a0b76993cf7ebb20f65face758dbe79b85567e5c5436"}
Sep 17 18:30:33.683594 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.683574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.683638 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.683613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60025e0d-4f71-4be8-bbfa-8665f8cba082-metrics-client-ca\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.683638 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.683630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-grpc-tls\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.683721 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.683699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdv4\" (UniqueName: \"kubernetes.io/projected/60025e0d-4f71-4be8-bbfa-8665f8cba082-kube-api-access-dhdv4\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.683797 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.683747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-tls\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.683797 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.683788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.683891 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.683836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.683891 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.683876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.685321 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.685079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60025e0d-4f71-4be8-bbfa-8665f8cba082-metrics-client-ca\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.685651 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.685606    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6497ddd774-twfmn" podStartSLOduration=1.343920581 podStartE2EDuration="5.685588741s" totalImagesPullingTime="4.34166816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:33.685257196 +0000 UTC m=+68.731660135" watchObservedRunningTime="2026-09-17 18:30:33.685588741 +0000 UTC m=+68.731991644"
Sep 17 18:30:33.686858 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.686815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.687234 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.686997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.687405 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.687381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.687476 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.687451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.687583 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.687566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-grpc-tls\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.688887 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.688871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/60025e0d-4f71-4be8-bbfa-8665f8cba082-secret-thanos-querier-tls\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.695308 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.695291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdv4\" (UniqueName: \"kubernetes.io/projected/60025e0d-4f71-4be8-bbfa-8665f8cba082-kube-api-access-dhdv4\") pod \"thanos-querier-6d659bf7cd-r78sr\" (UID: \"60025e0d-4f71-4be8-bbfa-8665f8cba082\") " pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:33.791094 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:33.791023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:34.072791 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:34.072654    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"]
Sep 17 18:30:34.683796 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:34.683681    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr" event={"ID":"60025e0d-4f71-4be8-bbfa-8665f8cba082","Type":"ContainerStarted","Data":"ba1830fe772fe05af96718d1a45c8368921ce42df1f5282ed7745ae799a6ef1f"}
Sep 17 18:30:34.686996 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:34.686922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x" event={"ID":"998c753e-fa09-4b9e-b81b-7deb8aaabfec","Type":"ContainerStarted","Data":"59cbb971b4ae80e85117ee93c90c1b9d9adab9de7945ab97e9a382f5a4705dc2"}
Sep 17 18:30:35.034419 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.034223    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-65fbc5879c-tlvdb"]
Sep 17 18:30:35.037408 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.037373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.041405 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.041349    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 18:30:35.041405 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.041364    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 18:30:35.041613 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.041597    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6k2kka26pc603\""
Sep 17 18:30:35.041684 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.041631    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gbmmn\""
Sep 17 18:30:35.041745 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.041731    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 18:30:35.041897 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.041823    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 18:30:35.048450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.048400    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65fbc5879c-tlvdb"]
Sep 17 18:30:35.097180 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.097148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/281b2b78-22d4-43b4-98f1-efbf3e0b140d-secret-metrics-server-client-certs\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.097311 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.097188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/281b2b78-22d4-43b4-98f1-efbf3e0b140d-audit-log\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.097311 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.097219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281b2b78-22d4-43b4-98f1-efbf3e0b140d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.097311 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.097249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cn8p\" (UniqueName: \"kubernetes.io/projected/281b2b78-22d4-43b4-98f1-efbf3e0b140d-kube-api-access-4cn8p\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.097311 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.097294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/281b2b78-22d4-43b4-98f1-efbf3e0b140d-secret-metrics-server-tls\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.097668 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.097530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/281b2b78-22d4-43b4-98f1-efbf3e0b140d-metrics-server-audit-profiles\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.097668 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.097573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/281b2b78-22d4-43b4-98f1-efbf3e0b140d-client-ca-bundle\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.166708 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.166673    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"]
Sep 17 18:30:35.171511 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.171469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:35.175548 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.175524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 18:30:35.175548 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.175966    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-45mqn\""
Sep 17 18:30:35.180017 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.178364    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"]
Sep 17 18:30:35.198775 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.198752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/281b2b78-22d4-43b4-98f1-efbf3e0b140d-client-ca-bundle\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.198969 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.198951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88749c0f-07ee-4e7c-97a3-c4afa2b1c78d-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-7c6x5\" (UID: \"88749c0f-07ee-4e7c-97a3-c4afa2b1c78d\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:35.199169 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.199153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/281b2b78-22d4-43b4-98f1-efbf3e0b140d-secret-metrics-server-client-certs\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.199378 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.199363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/281b2b78-22d4-43b4-98f1-efbf3e0b140d-audit-log\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.199954 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.199753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/281b2b78-22d4-43b4-98f1-efbf3e0b140d-audit-log\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.200033 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.199975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281b2b78-22d4-43b4-98f1-efbf3e0b140d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.200033 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.200013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cn8p\" (UniqueName: \"kubernetes.io/projected/281b2b78-22d4-43b4-98f1-efbf3e0b140d-kube-api-access-4cn8p\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.200133 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.200060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/281b2b78-22d4-43b4-98f1-efbf3e0b140d-secret-metrics-server-tls\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.200133 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.200094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw4lp\" (UniqueName: \"kubernetes.io/projected/88749c0f-07ee-4e7c-97a3-c4afa2b1c78d-kube-api-access-jw4lp\") pod \"monitoring-plugin-5984459ff8-7c6x5\" (UID: \"88749c0f-07ee-4e7c-97a3-c4afa2b1c78d\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:35.200218 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.200140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/281b2b78-22d4-43b4-98f1-efbf3e0b140d-metrics-server-audit-profiles\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.200851 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.200676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281b2b78-22d4-43b4-98f1-efbf3e0b140d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.201083 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.201062    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/281b2b78-22d4-43b4-98f1-efbf3e0b140d-metrics-server-audit-profiles\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.202667 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.202647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/281b2b78-22d4-43b4-98f1-efbf3e0b140d-secret-metrics-server-client-certs\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.203513 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.203472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/281b2b78-22d4-43b4-98f1-efbf3e0b140d-client-ca-bundle\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.204573 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.204529    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/281b2b78-22d4-43b4-98f1-efbf3e0b140d-secret-metrics-server-tls\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.214249 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.214216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cn8p\" (UniqueName: \"kubernetes.io/projected/281b2b78-22d4-43b4-98f1-efbf3e0b140d-kube-api-access-4cn8p\") pod \"metrics-server-65fbc5879c-tlvdb\" (UID: \"281b2b78-22d4-43b4-98f1-efbf3e0b140d\") " pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.301658 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.301577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw4lp\" (UniqueName: \"kubernetes.io/projected/88749c0f-07ee-4e7c-97a3-c4afa2b1c78d-kube-api-access-jw4lp\") pod \"monitoring-plugin-5984459ff8-7c6x5\" (UID: \"88749c0f-07ee-4e7c-97a3-c4afa2b1c78d\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:35.301799 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.301668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88749c0f-07ee-4e7c-97a3-c4afa2b1c78d-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-7c6x5\" (UID: \"88749c0f-07ee-4e7c-97a3-c4afa2b1c78d\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:35.301866 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:30:35.301796    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 18:30:35.301977 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:30:35.301882    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88749c0f-07ee-4e7c-97a3-c4afa2b1c78d-monitoring-plugin-cert podName:88749c0f-07ee-4e7c-97a3-c4afa2b1c78d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:35.801844562 +0000 UTC m=+70.848247448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/88749c0f-07ee-4e7c-97a3-c4afa2b1c78d-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-7c6x5" (UID: "88749c0f-07ee-4e7c-97a3-c4afa2b1c78d") : secret "monitoring-plugin-cert" not found
Sep 17 18:30:35.319635 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.319588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw4lp\" (UniqueName: \"kubernetes.io/projected/88749c0f-07ee-4e7c-97a3-c4afa2b1c78d-kube-api-access-jw4lp\") pod \"monitoring-plugin-5984459ff8-7c6x5\" (UID: \"88749c0f-07ee-4e7c-97a3-c4afa2b1c78d\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:35.351376 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.351312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:35.807863 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.807805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88749c0f-07ee-4e7c-97a3-c4afa2b1c78d-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-7c6x5\" (UID: \"88749c0f-07ee-4e7c-97a3-c4afa2b1c78d\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:35.817071 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.817008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88749c0f-07ee-4e7c-97a3-c4afa2b1c78d-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-7c6x5\" (UID: \"88749c0f-07ee-4e7c-97a3-c4afa2b1c78d\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:35.819379 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.817750    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-77dccf6945-6j4j7"]
Sep 17 18:30:35.820717 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.820695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:35.823322 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.823238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 18:30:35.823322 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.823237    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 18:30:35.823698 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.823639    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zdtnm\""
Sep 17 18:30:35.823698 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.823651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 18:30:35.823698 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.823666    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 18:30:35.827344 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.824964    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 18:30:35.831981 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.831946    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 18:30:35.838767 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.838743    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77dccf6945-6j4j7"]
Sep 17 18:30:35.909245 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.908881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-federate-client-tls\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:35.909245 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.908939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcabac82-0ab0-44a4-a296-451115c70543-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:35.909245 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.908990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:35.909245 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.909015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-secret-telemeter-client\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:35.909245 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.909057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcabac82-0ab0-44a4-a296-451115c70543-serving-certs-ca-bundle\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:35.909245 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.909089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcabac82-0ab0-44a4-a296-451115c70543-metrics-client-ca\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:35.909245 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.909116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-telemeter-client-tls\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:35.909245 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:35.909141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q5w6\" (UniqueName: \"kubernetes.io/projected/fcabac82-0ab0-44a4-a296-451115c70543-kube-api-access-6q5w6\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.011595 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.011131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-federate-client-tls\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.011595 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.011178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcabac82-0ab0-44a4-a296-451115c70543-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.011595 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.011247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.011595 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.011284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-secret-telemeter-client\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.011595 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.011336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcabac82-0ab0-44a4-a296-451115c70543-serving-certs-ca-bundle\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.011595 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.011367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcabac82-0ab0-44a4-a296-451115c70543-metrics-client-ca\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.011595 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.011397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-telemeter-client-tls\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.011595 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.011436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q5w6\" (UniqueName: \"kubernetes.io/projected/fcabac82-0ab0-44a4-a296-451115c70543-kube-api-access-6q5w6\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.013464 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.012176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcabac82-0ab0-44a4-a296-451115c70543-serving-certs-ca-bundle\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.013464 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.012794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcabac82-0ab0-44a4-a296-451115c70543-metrics-client-ca\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.013749 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.013691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcabac82-0ab0-44a4-a296-451115c70543-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.021195 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.021103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-telemeter-client-tls\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.022176 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.022081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q5w6\" (UniqueName: \"kubernetes.io/projected/fcabac82-0ab0-44a4-a296-451115c70543-kube-api-access-6q5w6\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.030665 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.029080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-federate-client-tls\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.030665 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.029149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-secret-telemeter-client\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.030665 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.029801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcabac82-0ab0-44a4-a296-451115c70543-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77dccf6945-6j4j7\" (UID: \"fcabac82-0ab0-44a4-a296-451115c70543\") " pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.090954 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.090871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:36.135314 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.135066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7"
Sep 17 18:30:36.202823 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.202753    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65fbc5879c-tlvdb"]
Sep 17 18:30:36.238709 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:36.238673    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281b2b78_22d4_43b4_98f1_efbf3e0b140d.slice/crio-ee515c41ff3fb2dfec96ddce6adbe2e4d277e687e44682b489eb442c8c3fb62d WatchSource:0}: Error finding container ee515c41ff3fb2dfec96ddce6adbe2e4d277e687e44682b489eb442c8c3fb62d: Status 404 returned error can't find the container with id ee515c41ff3fb2dfec96ddce6adbe2e4d277e687e44682b489eb442c8c3fb62d
Sep 17 18:30:36.405543 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.405402    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"]
Sep 17 18:30:36.489128 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.488422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77dccf6945-6j4j7"]
Sep 17 18:30:36.644356 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.644323    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tc2zd"
Sep 17 18:30:36.711394 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.711353    2577 generic.go:356] "Generic (PLEG): container finished" podID="72dd61e0-804b-4b85-abd1-eafbb220388f" containerID="00deaffdb230a37726835054115a0c4db27604532ba61ff163c0b6b7b8408c67" exitCode=0
Sep 17 18:30:36.711654 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.711509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rssq" event={"ID":"72dd61e0-804b-4b85-abd1-eafbb220388f","Type":"ContainerDied","Data":"00deaffdb230a37726835054115a0c4db27604532ba61ff163c0b6b7b8408c67"}
Sep 17 18:30:36.716132 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.716107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mqx5t" event={"ID":"656e881f-1982-4ce2-a57c-299fb552772b","Type":"ContainerStarted","Data":"e0563e5dad07d5e9afe1e9bb6eccc2eccee50639ffc63f8ba0d3a3eda48d40fa"}
Sep 17 18:30:36.716328 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.716259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mqx5t" event={"ID":"656e881f-1982-4ce2-a57c-299fb552772b","Type":"ContainerStarted","Data":"fce5ce0a840413596abc2344bbe2cb18fa724efd1808947583659495917986bd"}
Sep 17 18:30:36.721352 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.721329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x" event={"ID":"998c753e-fa09-4b9e-b81b-7deb8aaabfec","Type":"ContainerStarted","Data":"4aac45f26ce593bbee653c1c2f883b6e0a43273b01ce7ee64fdf030524411bac"}
Sep 17 18:30:36.727414 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.726736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx" event={"ID":"b3ba0a29-5045-4cf7-9008-998151191fb7","Type":"ContainerStarted","Data":"8cfa36d96a51ccdfb931242d07a3733f9946c2116134a907edbf166f578d1c7c"}
Sep 17 18:30:36.727414 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.726792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx" event={"ID":"b3ba0a29-5045-4cf7-9008-998151191fb7","Type":"ContainerStarted","Data":"e37f9a26b6eb83b3a12d0be7426680166c79bce076a8b43799701dffc1d35536"}
Sep 17 18:30:36.727414 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.726823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx" event={"ID":"b3ba0a29-5045-4cf7-9008-998151191fb7","Type":"ContainerStarted","Data":"236a64cb37c516156fe6d2cc305b38788cefb1a9809897704d998438da9719d8"}
Sep 17 18:30:36.729852 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.729796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb" event={"ID":"281b2b78-22d4-43b4-98f1-efbf3e0b140d","Type":"ContainerStarted","Data":"ee515c41ff3fb2dfec96ddce6adbe2e4d277e687e44682b489eb442c8c3fb62d"}
Sep 17 18:30:36.736024 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.735631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7" event={"ID":"fcabac82-0ab0-44a4-a296-451115c70543","Type":"ContainerStarted","Data":"d41523c29a0819b69f88ceb3519815fa52d9c5ae69aa380f5d4849e629f3bcb0"}
Sep 17 18:30:36.739613 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.739555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5" event={"ID":"88749c0f-07ee-4e7c-97a3-c4afa2b1c78d","Type":"ContainerStarted","Data":"7ec591dcb5a6bc5cef18fe0963553aabc4efee347f974a5d9e51e9a3c6052fa7"}
Sep 17 18:30:36.745401 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.744624    2577 generic.go:356] "Generic (PLEG): container finished" podID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerID="859e2be3fb6aa0a43d19313eb5a2d5ff48fbf8ef2b9dffcd3458906839831c76" exitCode=0
Sep 17 18:30:36.745401 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.744656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerDied","Data":"859e2be3fb6aa0a43d19313eb5a2d5ff48fbf8ef2b9dffcd3458906839831c76"}
Sep 17 18:30:36.797886 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.797080    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6wm5x" podStartSLOduration=4.584559106 podStartE2EDuration="6.79706052s" totalImagesPullingTime="2.212501414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:36.796137962 +0000 UTC m=+71.842540888" watchObservedRunningTime="2026-09-17 18:30:36.79706052 +0000 UTC m=+71.843463422"
Sep 17 18:30:36.797886 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.797570    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mqx5t" podStartSLOduration=67.499717095 podStartE2EDuration="1m9.797558557s" totalImagesPullingTime="2.297841462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:36.767666892 +0000 UTC m=+71.814069817" watchObservedRunningTime="2026-09-17 18:30:36.797558557 +0000 UTC m=+71.843961462"
Sep 17 18:30:36.829624 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:36.829430    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-585cx" podStartSLOduration=4.45653714 podStartE2EDuration="6.829409519s" totalImagesPullingTime="2.372872379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:36.827944223 +0000 UTC m=+71.874347162" watchObservedRunningTime="2026-09-17 18:30:36.829409519 +0000 UTC m=+71.875812420"
Sep 17 18:30:37.757247 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:37.757205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rssq" event={"ID":"72dd61e0-804b-4b85-abd1-eafbb220388f","Type":"ContainerStarted","Data":"e3dbda2e6fe54157d4db9772c8ff1be2eaad533f5cb46ba9d9fdf210b3459926"}
Sep 17 18:30:37.757424 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:37.757273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rssq" event={"ID":"72dd61e0-804b-4b85-abd1-eafbb220388f","Type":"ContainerStarted","Data":"22b32b913825329fbfb1d62ac1b2f8f725605ece6640d4fe9cf03dceb2f40b8e"}
Sep 17 18:30:37.782991 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:37.780594    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7rssq" podStartSLOduration=5.079485711 podStartE2EDuration="7.780572559s" totalImagesPullingTime="2.575613658s" totalInitContainerRuntime="125.47319ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:37.779985298 +0000 UTC m=+72.826388218" watchObservedRunningTime="2026-09-17 18:30:37.780572559 +0000 UTC m=+72.826975460"
Sep 17 18:30:38.292346 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.292313    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d9f568fb6-7n2l2"]
Sep 17 18:30:38.294954 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.294933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.303767 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.303744    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 18:30:38.308194 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.308169    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9f568fb6-7n2l2"]
Sep 17 18:30:38.344732 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.344699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-console-config\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.344732 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.344746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-trusted-ca-bundle\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.344983 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.344772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-service-ca\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.344983 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.344850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-oauth-serving-cert\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.344983 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.344885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-oauth-config\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.344983 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.344956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt7zq\" (UniqueName: \"kubernetes.io/projected/1533d684-b049-47f2-b591-81b31562f93e-kube-api-access-kt7zq\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.345140 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.345009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-serving-cert\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.446094 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.446054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-console-config\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.446303 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.446125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-trusted-ca-bundle\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.446303 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.446167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-service-ca\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.446303 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.446199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-oauth-serving-cert\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.446303 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.446220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-oauth-config\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.446303 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.446258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt7zq\" (UniqueName: \"kubernetes.io/projected/1533d684-b049-47f2-b591-81b31562f93e-kube-api-access-kt7zq\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.446303 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.446301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-serving-cert\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.447203 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.446890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-service-ca\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.447203 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.446920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-console-config\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.447203 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.447122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-trusted-ca-bundle\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.447429 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.447368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-oauth-serving-cert\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.449242 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.449221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-serving-cert\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.449365 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.449341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-oauth-config\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.456213 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.456169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt7zq\" (UniqueName: \"kubernetes.io/projected/1533d684-b049-47f2-b591-81b31562f93e-kube-api-access-kt7zq\") pod \"console-d9f568fb6-7n2l2\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") " pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.605703 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.605616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:38.671454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.671412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:38.671619 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.671487    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:38.677098 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.677072    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:38.764862 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:38.764833    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:30:41.669875 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.669797    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9f568fb6-7n2l2"]
Sep 17 18:30:41.775415 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.775377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j94pq" event={"ID":"ebc60a41-a81e-404d-978f-102cc369618b","Type":"ContainerStarted","Data":"bdb837546e5d0ed75726f4b89f2b0461cd00e0b8f5e435c1491c62dac495af9a"}
Sep 17 18:30:41.777207 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.777175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cx897" event={"ID":"21c07112-ca6a-4083-9fad-f0b5b5afb0ba","Type":"ContainerStarted","Data":"26f18cc65673ff8136cc3348d752373e75ab47288c08d50dce1ae8e3221ed2ef"}
Sep 17 18:30:41.777351 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.777230    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:30:41.779308 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.779280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb" event={"ID":"281b2b78-22d4-43b4-98f1-efbf3e0b140d","Type":"ContainerStarted","Data":"9056b17bc2e51bb8f6c2447178b00098ad7918e12107ebcaf644feab40573dc7"}
Sep 17 18:30:41.781548 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.781522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr" event={"ID":"60025e0d-4f71-4be8-bbfa-8665f8cba082","Type":"ContainerStarted","Data":"1a64efc39f57af5b8b7fdcfa02c0ca9d802042b2976acf7dd4a7ffaf4be1231f"}
Sep 17 18:30:41.781651 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.781591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr" event={"ID":"60025e0d-4f71-4be8-bbfa-8665f8cba082","Type":"ContainerStarted","Data":"0c115ce6d1a8d200285a8666ff99fdbe666055de7bc8dbdf4b5533b8a323c6f5"}
Sep 17 18:30:41.783675 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.783654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7" event={"ID":"fcabac82-0ab0-44a4-a296-451115c70543","Type":"ContainerStarted","Data":"0b36da689e4d168d75141b26a410ce31e9f391686a8d771f9034bea6f5075b32"}
Sep 17 18:30:41.783764 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.783703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7" event={"ID":"fcabac82-0ab0-44a4-a296-451115c70543","Type":"ContainerStarted","Data":"f3a3f48209739c41aeb4991f5441b11cb0405d7674d2ff17efd83d7275627531"}
Sep 17 18:30:41.785277 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.785256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5" event={"ID":"88749c0f-07ee-4e7c-97a3-c4afa2b1c78d","Type":"ContainerStarted","Data":"cb7765218e390e1f8440bcf0cc57f7e1dd6fd9e4c22be02d4ff9a451f1ebe83c"}
Sep 17 18:30:41.785478 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.785461    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:41.792762 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.792720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5"
Sep 17 18:30:41.793363 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.793324    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j94pq" podStartSLOduration=66.970044667 podStartE2EDuration="1m14.793311046s" totalImagesPullingTime="7.823266379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:41.792846564 +0000 UTC m=+76.839249466" watchObservedRunningTime="2026-09-17 18:30:41.793311046 +0000 UTC m=+76.839713940"
Sep 17 18:30:41.812917 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.812864    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb" podStartSLOduration=1.652798864 podStartE2EDuration="6.812847088s" totalImagesPullingTime="5.160048224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:41.812095545 +0000 UTC m=+76.858498477" watchObservedRunningTime="2026-09-17 18:30:41.812847088 +0000 UTC m=+76.859249989"
Sep 17 18:30:41.829176 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.829127    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cx897" podStartSLOduration=67.022276338 podStartE2EDuration="1m14.828977561s" totalImagesPullingTime="7.806701223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:41.828604817 +0000 UTC m=+76.875007737" watchObservedRunningTime="2026-09-17 18:30:41.828977561 +0000 UTC m=+76.875380458"
Sep 17 18:30:41.846388 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:41.846349    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-7c6x5" podStartSLOduration=1.858444143 podStartE2EDuration="6.846333473s" totalImagesPullingTime="4.98788933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:41.845115194 +0000 UTC m=+76.891518111" watchObservedRunningTime="2026-09-17 18:30:41.846333473 +0000 UTC m=+76.892736378"
Sep 17 18:30:41.977554 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:41.977523    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1533d684_b049_47f2_b591_81b31562f93e.slice/crio-fac05dc924cc355aa8d4934ad7c18b787ee4b329bc4d1d369dc0f98b865b9cfd WatchSource:0}: Error finding container fac05dc924cc355aa8d4934ad7c18b787ee4b329bc4d1d369dc0f98b865b9cfd: Status 404 returned error can't find the container with id fac05dc924cc355aa8d4934ad7c18b787ee4b329bc4d1d369dc0f98b865b9cfd
Sep 17 18:30:42.791968 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.791935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7" event={"ID":"fcabac82-0ab0-44a4-a296-451115c70543","Type":"ContainerStarted","Data":"304281a278338ab35829e260c730b087cb5711de7e0ddb43407cabb1fea39a4d"}
Sep 17 18:30:42.794654 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.794630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerStarted","Data":"a52e851ea45a5193b0384521167f75128b905dcce26968179deac2be034fc86e"}
Sep 17 18:30:42.794772 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.794689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerStarted","Data":"251c48d9126f6838cb5cf996a6711c7c20ea0eab03683825813ff851f93f20a7"}
Sep 17 18:30:42.794772 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.794723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerStarted","Data":"69fc7b2b63e341d4c0281d9a2f74a605ed9ebe85979e5d5c2c8530da61803095"}
Sep 17 18:30:42.794772 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.794761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerStarted","Data":"7249c34083f1c447ec3cfa737b19674c50b6b9cf583b290cd2c31b85ebb7cf95"}
Sep 17 18:30:42.794876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.794785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerStarted","Data":"92eec66fb90e7b9043201d164d391a36df4d5e37fdbf2c57f4b5cd919b59d180"}
Sep 17 18:30:42.796135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.796112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9f568fb6-7n2l2" event={"ID":"1533d684-b049-47f2-b591-81b31562f93e","Type":"ContainerStarted","Data":"b5b8c9f7a72487f5c507a72f3eaa488e920c6e75c4e0e1a6c58e9481ea140814"}
Sep 17 18:30:42.796252 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.796154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9f568fb6-7n2l2" event={"ID":"1533d684-b049-47f2-b591-81b31562f93e","Type":"ContainerStarted","Data":"fac05dc924cc355aa8d4934ad7c18b787ee4b329bc4d1d369dc0f98b865b9cfd"}
Sep 17 18:30:42.798781 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.798754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr" event={"ID":"60025e0d-4f71-4be8-bbfa-8665f8cba082","Type":"ContainerStarted","Data":"d7685efd9c8999600d70bea371ef02cd6933f6225cdc800a1a38c9f9841c25d2"}
Sep 17 18:30:42.817488 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.817444    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-77dccf6945-6j4j7" podStartSLOduration=2.910415788 podStartE2EDuration="7.817431102s" totalImagesPullingTime="4.907015314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:42.816189149 +0000 UTC m=+77.862592074" watchObservedRunningTime="2026-09-17 18:30:42.817431102 +0000 UTC m=+77.863834002"
Sep 17 18:30:42.838770 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:42.838730    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d9f568fb6-7n2l2" podStartSLOduration=4.838716302 podStartE2EDuration="4.838716302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:42.83836364 +0000 UTC m=+77.884766543" watchObservedRunningTime="2026-09-17 18:30:42.838716302 +0000 UTC m=+77.885119203"
Sep 17 18:30:43.807198 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:43.807161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerStarted","Data":"4fa976adacd46987b6ed94740a46d577cbcf5ac9fa7b5fd726c815a5a6a49423"}
Sep 17 18:30:43.812159 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:43.812125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr" event={"ID":"60025e0d-4f71-4be8-bbfa-8665f8cba082","Type":"ContainerStarted","Data":"65c908c7c5e5882061353d2037cf1b9ed0a33cbe0b6fa9b6bf0ef7c7b02eab14"}
Sep 17 18:30:43.812296 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:43.812238    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr" event={"ID":"60025e0d-4f71-4be8-bbfa-8665f8cba082","Type":"ContainerStarted","Data":"c057cd41a9ce7fffb1f0af8be11474f1661245032e4c96917cad459698984208"}
Sep 17 18:30:43.812296 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:43.812287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr" event={"ID":"60025e0d-4f71-4be8-bbfa-8665f8cba082","Type":"ContainerStarted","Data":"553a8b737d9dd286a2f9f3e6e6025672cb5f36fe3dc0b8ca75e897356dad2c3a"}
Sep 17 18:30:43.839461 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:43.839411    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.40249498 podStartE2EDuration="12.839393655s" totalImagesPullingTime="8.405037784s" totalInitContainerRuntime="31.860891ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:43.83689261 +0000 UTC m=+78.883295559" watchObservedRunningTime="2026-09-17 18:30:43.839393655 +0000 UTC m=+78.885796556"
Sep 17 18:30:43.859948 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:43.859876    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr" podStartSLOduration=2.79677972 podStartE2EDuration="10.859858592s" totalImagesPullingTime="8.063078872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:43.859645382 +0000 UTC m=+78.906048284" watchObservedRunningTime="2026-09-17 18:30:43.859858592 +0000 UTC m=+78.906261492"
Sep 17 18:30:44.628370 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:44.628341    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cff9d988d-5fl5c"
Sep 17 18:30:44.816283 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:44.816252    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:46.742201 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.742153    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d9f568fb6-7n2l2"]
Sep 17 18:30:46.776577 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.776550    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bf956847b-fg2mj"]
Sep 17 18:30:46.778847 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.778832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:46.792583 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.792558    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf956847b-fg2mj"]
Sep 17 18:30:46.933304 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.933268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-trusted-ca-bundle\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:46.933304 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.933310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljk5g\" (UniqueName: \"kubernetes.io/projected/7c689fe4-4f75-4802-b2c9-830990295a35-kube-api-access-ljk5g\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:46.933489 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.933372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-serving-cert\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:46.933489 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.933415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-service-ca\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:46.933489 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.933446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-oauth-config\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:46.933612 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.933524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-console-config\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:46.933612 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:46.933553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-oauth-serving-cert\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.034788 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.034716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-console-config\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.034788 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.034750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-oauth-serving-cert\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.034788 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.034782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-trusted-ca-bundle\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.035042 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.034814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljk5g\" (UniqueName: \"kubernetes.io/projected/7c689fe4-4f75-4802-b2c9-830990295a35-kube-api-access-ljk5g\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.035042 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.034858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-serving-cert\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.035042 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.034921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-service-ca\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.035042 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.034965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-oauth-config\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.035553 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.035520    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-console-config\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.035669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.035644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-oauth-serving-cert\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.035669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.035663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-service-ca\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.035801 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.035711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-trusted-ca-bundle\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.037369 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.037340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-serving-cert\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.037446 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.037417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-oauth-config\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.057069 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.057040    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljk5g\" (UniqueName: \"kubernetes.io/projected/7c689fe4-4f75-4802-b2c9-830990295a35-kube-api-access-ljk5g\") pod \"console-7bf956847b-fg2mj\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") " pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.088096 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.088071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:47.369258 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.369188    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf956847b-fg2mj"]
Sep 17 18:30:47.372733 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:30:47.372712    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c689fe4_4f75_4802_b2c9_830990295a35.slice/crio-f9d365df704a43a2b8edb9509a031cfdb21ec76476080aee9f8a4193928a445f WatchSource:0}: Error finding container f9d365df704a43a2b8edb9509a031cfdb21ec76476080aee9f8a4193928a445f: Status 404 returned error can't find the container with id f9d365df704a43a2b8edb9509a031cfdb21ec76476080aee9f8a4193928a445f
Sep 17 18:30:47.829113 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.829079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf956847b-fg2mj" event={"ID":"7c689fe4-4f75-4802-b2c9-830990295a35","Type":"ContainerStarted","Data":"b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447"}
Sep 17 18:30:47.829437 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.829133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf956847b-fg2mj" event={"ID":"7c689fe4-4f75-4802-b2c9-830990295a35","Type":"ContainerStarted","Data":"f9d365df704a43a2b8edb9509a031cfdb21ec76476080aee9f8a4193928a445f"}
Sep 17 18:30:47.849555 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:47.849506    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bf956847b-fg2mj" podStartSLOduration=1.849492611 podStartE2EDuration="1.849492611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:47.846965248 +0000 UTC m=+82.893368147" watchObservedRunningTime="2026-09-17 18:30:47.849492611 +0000 UTC m=+82.895895511"
Sep 17 18:30:48.606424 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:48.606389    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:30:50.826477 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:50.826441    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d659bf7cd-r78sr"
Sep 17 18:30:55.352427 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:55.352395    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:55.352855 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:55.352447    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:30:57.088985 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:57.088949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:57.089399 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:57.088998    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:57.093702 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:57.093676    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:57.878245 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:57.878215    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:30:57.937893 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:30:57.937858    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6497ddd774-twfmn"]
Sep 17 18:31:01.168545 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:01.168513    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rlt2p_0885b1df-0695-4589-8ae5-387f4a25c048/serve-healthcheck-canary/0.log"
Sep 17 18:31:11.761102 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:11.761039    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d9f568fb6-7n2l2" podUID="1533d684-b049-47f2-b591-81b31562f93e" containerName="console" containerID="cri-o://b5b8c9f7a72487f5c507a72f3eaa488e920c6e75c4e0e1a6c58e9481ea140814" gracePeriod=15
Sep 17 18:31:11.937233 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:11.937195    2577 generic.go:356] "Generic (PLEG): container finished" podID="1533d684-b049-47f2-b591-81b31562f93e" containerID="b5b8c9f7a72487f5c507a72f3eaa488e920c6e75c4e0e1a6c58e9481ea140814" exitCode=0
Sep 17 18:31:11.937407 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:11.937265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9f568fb6-7n2l2" event={"ID":"1533d684-b049-47f2-b591-81b31562f93e","Type":"ContainerDied","Data":"b5b8c9f7a72487f5c507a72f3eaa488e920c6e75c4e0e1a6c58e9481ea140814"}
Sep 17 18:31:12.051759 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.051731    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:31:12.126226 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.126191    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-serving-cert\") pod \"1533d684-b049-47f2-b591-81b31562f93e\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") "
Sep 17 18:31:12.126403 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.126237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-service-ca\") pod \"1533d684-b049-47f2-b591-81b31562f93e\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") "
Sep 17 18:31:12.126403 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.126276    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-console-config\") pod \"1533d684-b049-47f2-b591-81b31562f93e\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") "
Sep 17 18:31:12.126403 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.126312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-oauth-serving-cert\") pod \"1533d684-b049-47f2-b591-81b31562f93e\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") "
Sep 17 18:31:12.126403 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.126339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-oauth-config\") pod \"1533d684-b049-47f2-b591-81b31562f93e\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") "
Sep 17 18:31:12.126403 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.126364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt7zq\" (UniqueName: \"kubernetes.io/projected/1533d684-b049-47f2-b591-81b31562f93e-kube-api-access-kt7zq\") pod \"1533d684-b049-47f2-b591-81b31562f93e\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") "
Sep 17 18:31:12.126403 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.126398    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-trusted-ca-bundle\") pod \"1533d684-b049-47f2-b591-81b31562f93e\" (UID: \"1533d684-b049-47f2-b591-81b31562f93e\") "
Sep 17 18:31:12.126846 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.126808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1533d684-b049-47f2-b591-81b31562f93e" (UID: "1533d684-b049-47f2-b591-81b31562f93e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:12.126927 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.126824    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-service-ca" (OuterVolumeSpecName: "service-ca") pod "1533d684-b049-47f2-b591-81b31562f93e" (UID: "1533d684-b049-47f2-b591-81b31562f93e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:12.127084 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.126969    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1533d684-b049-47f2-b591-81b31562f93e" (UID: "1533d684-b049-47f2-b591-81b31562f93e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:12.127216 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.127130    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-console-config" (OuterVolumeSpecName: "console-config") pod "1533d684-b049-47f2-b591-81b31562f93e" (UID: "1533d684-b049-47f2-b591-81b31562f93e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:12.128813 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.128787    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1533d684-b049-47f2-b591-81b31562f93e" (UID: "1533d684-b049-47f2-b591-81b31562f93e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:12.128962 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.128938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1533d684-b049-47f2-b591-81b31562f93e" (UID: "1533d684-b049-47f2-b591-81b31562f93e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:12.129294 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.129275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1533d684-b049-47f2-b591-81b31562f93e-kube-api-access-kt7zq" (OuterVolumeSpecName: "kube-api-access-kt7zq") pod "1533d684-b049-47f2-b591-81b31562f93e" (UID: "1533d684-b049-47f2-b591-81b31562f93e"). InnerVolumeSpecName "kube-api-access-kt7zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:12.227350 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.227318    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt7zq\" (UniqueName: \"kubernetes.io/projected/1533d684-b049-47f2-b591-81b31562f93e-kube-api-access-kt7zq\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:12.227350 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.227347    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-trusted-ca-bundle\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:12.227350 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.227356    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-serving-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:12.227561 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.227365    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-service-ca\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:12.227561 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.227373    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-console-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:12.227561 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.227382    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1533d684-b049-47f2-b591-81b31562f93e-oauth-serving-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:12.227561 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.227391    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1533d684-b049-47f2-b591-81b31562f93e-console-oauth-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:12.801600 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.801572    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cx897"
Sep 17 18:31:12.943360 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.943331    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9f568fb6-7n2l2"
Sep 17 18:31:12.943531 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.943329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9f568fb6-7n2l2" event={"ID":"1533d684-b049-47f2-b591-81b31562f93e","Type":"ContainerDied","Data":"fac05dc924cc355aa8d4934ad7c18b787ee4b329bc4d1d369dc0f98b865b9cfd"}
Sep 17 18:31:12.943531 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.943462    2577 scope.go:117] "RemoveContainer" containerID="b5b8c9f7a72487f5c507a72f3eaa488e920c6e75c4e0e1a6c58e9481ea140814"
Sep 17 18:31:12.971015 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.970982    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d9f568fb6-7n2l2"]
Sep 17 18:31:12.973741 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:12.973713    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d9f568fb6-7n2l2"]
Sep 17 18:31:13.238122 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:13.238091    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1533d684-b049-47f2-b591-81b31562f93e" path="/var/lib/kubelet/pods/1533d684-b049-47f2-b591-81b31562f93e/volumes"
Sep 17 18:31:15.356581 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:15.356552    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:31:15.360318 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:15.360297    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-65fbc5879c-tlvdb"
Sep 17 18:31:22.959000 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:22.958946    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6497ddd774-twfmn" podUID="89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" containerName="console" containerID="cri-o://da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4" gracePeriod=15
Sep 17 18:31:23.196444 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.196419    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:31:23.214124 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.214067    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-service-ca\") pod \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") "
Sep 17 18:31:23.214124 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.214103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vv7q\" (UniqueName: \"kubernetes.io/projected/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-kube-api-access-2vv7q\") pod \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") "
Sep 17 18:31:23.214249 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.214129    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-oauth-config\") pod \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") "
Sep 17 18:31:23.214249 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.214182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-config\") pod \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") "
Sep 17 18:31:23.214249 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.214231    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-oauth-serving-cert\") pod \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") "
Sep 17 18:31:23.214409 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.214285    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-serving-cert\") pod \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\" (UID: \"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3\") "
Sep 17 18:31:23.214643 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.214593    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-service-ca" (OuterVolumeSpecName: "service-ca") pod "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" (UID: "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:23.214643 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.214590    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-config" (OuterVolumeSpecName: "console-config") pod "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" (UID: "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:23.214790 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.214684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" (UID: "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:23.216537 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.216507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-kube-api-access-2vv7q" (OuterVolumeSpecName: "kube-api-access-2vv7q") pod "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" (UID: "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3"). InnerVolumeSpecName "kube-api-access-2vv7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:23.217236 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.217196    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" (UID: "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:23.217962 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.216928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" (UID: "89e0d2c8-8c39-4b9b-9ae4-4512e86406b3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:23.315327 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.315295    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-serving-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.315327 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.315321    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-service-ca\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.315327 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.315331    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vv7q\" (UniqueName: \"kubernetes.io/projected/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-kube-api-access-2vv7q\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.315550 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.315341    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-oauth-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.315550 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.315350    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-console-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.315550 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.315358    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3-oauth-serving-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.988479 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.988442    2577 generic.go:356] "Generic (PLEG): container finished" podID="89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" containerID="da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4" exitCode=0
Sep 17 18:31:23.988949 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.988484    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6497ddd774-twfmn"
Sep 17 18:31:23.988949 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.988512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6497ddd774-twfmn" event={"ID":"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3","Type":"ContainerDied","Data":"da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4"}
Sep 17 18:31:23.988949 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.988556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6497ddd774-twfmn" event={"ID":"89e0d2c8-8c39-4b9b-9ae4-4512e86406b3","Type":"ContainerDied","Data":"38fa5706c29771219e40f164747f59f9100d4a50a60a524fca0a754afb34124e"}
Sep 17 18:31:23.988949 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.988575    2577 scope.go:117] "RemoveContainer" containerID="da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4"
Sep 17 18:31:23.996706 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.996687    2577 scope.go:117] "RemoveContainer" containerID="da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4"
Sep 17 18:31:23.997035 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:31:23.997014    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4\": container with ID starting with da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4 not found: ID does not exist" containerID="da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4"
Sep 17 18:31:23.997087 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:23.997047    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4"} err="failed to get container status \"da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4\": rpc error: code = NotFound desc = could not find container \"da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4\": container with ID starting with da281fc2b71c77dcb7453aef5329d12eaacf3541cf3026334446f3e3cafb7bc4 not found: ID does not exist"
Sep 17 18:31:24.005018 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:24.004989    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6497ddd774-twfmn"]
Sep 17 18:31:24.008545 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:24.008518    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6497ddd774-twfmn"]
Sep 17 18:31:25.238334 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:25.238287    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" path="/var/lib/kubelet/pods/89e0d2c8-8c39-4b9b-9ae4-4512e86406b3/volumes"
Sep 17 18:31:50.940193 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:50.940155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:31:50.940752 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:50.940459    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="alertmanager" containerID="cri-o://92eec66fb90e7b9043201d164d391a36df4d5e37fdbf2c57f4b5cd919b59d180" gracePeriod=120
Sep 17 18:31:50.940752 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:50.940533    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy-metric" containerID="cri-o://a52e851ea45a5193b0384521167f75128b905dcce26968179deac2be034fc86e" gracePeriod=120
Sep 17 18:31:50.940752 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:50.940568    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy-web" containerID="cri-o://69fc7b2b63e341d4c0281d9a2f74a605ed9ebe85979e5d5c2c8530da61803095" gracePeriod=120
Sep 17 18:31:50.940752 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:50.940604    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="prom-label-proxy" containerID="cri-o://4fa976adacd46987b6ed94740a46d577cbcf5ac9fa7b5fd726c815a5a6a49423" gracePeriod=120
Sep 17 18:31:50.940752 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:50.940558    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="config-reloader" containerID="cri-o://7249c34083f1c447ec3cfa737b19674c50b6b9cf583b290cd2c31b85ebb7cf95" gracePeriod=120
Sep 17 18:31:50.940752 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:50.940654    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy" containerID="cri-o://251c48d9126f6838cb5cf996a6711c7c20ea0eab03683825813ff851f93f20a7" gracePeriod=120
Sep 17 18:31:51.098632 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:51.098603    2577 generic.go:356] "Generic (PLEG): container finished" podID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerID="4fa976adacd46987b6ed94740a46d577cbcf5ac9fa7b5fd726c815a5a6a49423" exitCode=0
Sep 17 18:31:51.098632 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:51.098627    2577 generic.go:356] "Generic (PLEG): container finished" podID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerID="251c48d9126f6838cb5cf996a6711c7c20ea0eab03683825813ff851f93f20a7" exitCode=0
Sep 17 18:31:51.098632 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:51.098633    2577 generic.go:356] "Generic (PLEG): container finished" podID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerID="7249c34083f1c447ec3cfa737b19674c50b6b9cf583b290cd2c31b85ebb7cf95" exitCode=0
Sep 17 18:31:51.098843 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:51.098641    2577 generic.go:356] "Generic (PLEG): container finished" podID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerID="92eec66fb90e7b9043201d164d391a36df4d5e37fdbf2c57f4b5cd919b59d180" exitCode=0
Sep 17 18:31:51.098843 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:51.098671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerDied","Data":"4fa976adacd46987b6ed94740a46d577cbcf5ac9fa7b5fd726c815a5a6a49423"}
Sep 17 18:31:51.098843 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:51.098737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerDied","Data":"251c48d9126f6838cb5cf996a6711c7c20ea0eab03683825813ff851f93f20a7"}
Sep 17 18:31:51.098843 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:51.098765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerDied","Data":"7249c34083f1c447ec3cfa737b19674c50b6b9cf583b290cd2c31b85ebb7cf95"}
Sep 17 18:31:51.098843 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:51.098789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerDied","Data":"92eec66fb90e7b9043201d164d391a36df4d5e37fdbf2c57f4b5cd919b59d180"}
Sep 17 18:31:52.107402 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.107373    2577 generic.go:356] "Generic (PLEG): container finished" podID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerID="a52e851ea45a5193b0384521167f75128b905dcce26968179deac2be034fc86e" exitCode=0
Sep 17 18:31:52.107402 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.107397    2577 generic.go:356] "Generic (PLEG): container finished" podID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerID="69fc7b2b63e341d4c0281d9a2f74a605ed9ebe85979e5d5c2c8530da61803095" exitCode=0
Sep 17 18:31:52.107743 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.107452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerDied","Data":"a52e851ea45a5193b0384521167f75128b905dcce26968179deac2be034fc86e"}
Sep 17 18:31:52.107743 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.107544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerDied","Data":"69fc7b2b63e341d4c0281d9a2f74a605ed9ebe85979e5d5c2c8530da61803095"}
Sep 17 18:31:52.185591 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.185567    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:52.369608 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369526    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.369608 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369566    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-cluster-tls-config\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.369608 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369598    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-main-db\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.369862 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-trusted-ca-bundle\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.369862 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369650    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-main-tls\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.369862 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369675    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.369862 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369714    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-out\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.369862 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369739    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-metrics-client-ca\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.369862 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369768    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rxhj\" (UniqueName: \"kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-kube-api-access-2rxhj\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.369862 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-volume\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.369862 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-web-config\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.370284 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.370284 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.369934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-tls-assets\") pod \"29e59426-8fd1-4741-a6c2-dbd7300641a2\" (UID: \"29e59426-8fd1-4741-a6c2-dbd7300641a2\") "
Sep 17 18:31:52.370284 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.370008    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:52.370284 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.370070    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:52.370529 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.370484    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-main-db\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.370529 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.370514    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.371299 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.371267    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:52.373610 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.373573    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:52.373764 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.373745    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:52.373839 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.373783    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:52.374085 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.374048    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:52.374228 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.374194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-out" (OuterVolumeSpecName: "config-out") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:52.374349 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.374215    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:52.374415 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.374374    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-volume" (OuterVolumeSpecName: "config-volume") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:52.374468 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.374445    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-kube-api-access-2rxhj" (OuterVolumeSpecName: "kube-api-access-2rxhj") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "kube-api-access-2rxhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:52.377564 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.377536    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:52.384214 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.384194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-web-config" (OuterVolumeSpecName: "web-config") pod "29e59426-8fd1-4741-a6c2-dbd7300641a2" (UID: "29e59426-8fd1-4741-a6c2-dbd7300641a2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:52.471565 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471533    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-main-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.471565 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471564    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.471565 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471574    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-out\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.471714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471583    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29e59426-8fd1-4741-a6c2-dbd7300641a2-metrics-client-ca\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.471714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471591    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rxhj\" (UniqueName: \"kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-kube-api-access-2rxhj\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.471714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471599    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-config-volume\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.471714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471606    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-web-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.471714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471615    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.471714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471623    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29e59426-8fd1-4741-a6c2-dbd7300641a2-tls-assets\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.471714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471631    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:52.471714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:52.471640    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29e59426-8fd1-4741-a6c2-dbd7300641a2-cluster-tls-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:31:53.116637 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.116599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29e59426-8fd1-4741-a6c2-dbd7300641a2","Type":"ContainerDied","Data":"90b3164bf55798464637a3f9d418128eedb0795dba036618c702006ec05aed92"}
Sep 17 18:31:53.117017 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.116680    2577 scope.go:117] "RemoveContainer" containerID="4fa976adacd46987b6ed94740a46d577cbcf5ac9fa7b5fd726c815a5a6a49423"
Sep 17 18:31:53.117017 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.116619    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.124086 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.124044    2577 scope.go:117] "RemoveContainer" containerID="a52e851ea45a5193b0384521167f75128b905dcce26968179deac2be034fc86e"
Sep 17 18:31:53.130691 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.130676    2577 scope.go:117] "RemoveContainer" containerID="251c48d9126f6838cb5cf996a6711c7c20ea0eab03683825813ff851f93f20a7"
Sep 17 18:31:53.137500 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.137482    2577 scope.go:117] "RemoveContainer" containerID="69fc7b2b63e341d4c0281d9a2f74a605ed9ebe85979e5d5c2c8530da61803095"
Sep 17 18:31:53.139388 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.139367    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:31:53.144876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.144851    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:31:53.149625 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.149590    2577 scope.go:117] "RemoveContainer" containerID="7249c34083f1c447ec3cfa737b19674c50b6b9cf583b290cd2c31b85ebb7cf95"
Sep 17 18:31:53.157823 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.157789    2577 scope.go:117] "RemoveContainer" containerID="92eec66fb90e7b9043201d164d391a36df4d5e37fdbf2c57f4b5cd919b59d180"
Sep 17 18:31:53.165282 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.165244    2577 scope.go:117] "RemoveContainer" containerID="859e2be3fb6aa0a43d19313eb5a2d5ff48fbf8ef2b9dffcd3458906839831c76"
Sep 17 18:31:53.174623 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.174603    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:31:53.175045 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175027    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="config-reloader"
Sep 17 18:31:53.175126 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175049    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="config-reloader"
Sep 17 18:31:53.175172 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175156    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="prom-label-proxy"
Sep 17 18:31:53.175172 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175168    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="prom-label-proxy"
Sep 17 18:31:53.175256 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175182    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy-metric"
Sep 17 18:31:53.175256 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175188    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy-metric"
Sep 17 18:31:53.175256 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175200    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1533d684-b049-47f2-b591-81b31562f93e" containerName="console"
Sep 17 18:31:53.175256 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175209    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1533d684-b049-47f2-b591-81b31562f93e" containerName="console"
Sep 17 18:31:53.175256 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175224    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" containerName="console"
Sep 17 18:31:53.175256 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175231    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" containerName="console"
Sep 17 18:31:53.175256 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175241    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="init-config-reloader"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175249    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="init-config-reloader"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175298    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="alertmanager"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175303    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="alertmanager"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175310    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy-web"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175315    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy-web"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175320    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175326    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175393    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89e0d2c8-8c39-4b9b-9ae4-4512e86406b3" containerName="console"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175400    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1533d684-b049-47f2-b591-81b31562f93e" containerName="console"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175406    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="alertmanager"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175413    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy-web"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175419    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175424    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="config-reloader"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175430    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="prom-label-proxy"
Sep 17 18:31:53.175454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.175436    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" containerName="kube-rbac-proxy-metric"
Sep 17 18:31:53.184631 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.184612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.187234 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.187212    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 18:31:53.187551 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.187534    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 18:31:53.188010 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.187991    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 18:31:53.188094 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.188061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zlj98\""
Sep 17 18:31:53.188148 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.188127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 18:31:53.188148 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.188134    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 18:31:53.188258 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.188139    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 18:31:53.188258 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.188061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 18:31:53.188365 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.188348    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 18:31:53.193689 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.193667    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 18:31:53.200768 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.200743    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:31:53.238550 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.238524    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e59426-8fd1-4741-a6c2-dbd7300641a2" path="/var/lib/kubelet/pods/29e59426-8fd1-4741-a6c2-dbd7300641a2/volumes"
Sep 17 18:31:53.278196 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3267dea6-9e40-494a-aeaa-458e00956620-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278196 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3267dea6-9e40-494a-aeaa-458e00956620-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278196 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3267dea6-9e40-494a-aeaa-458e00956620-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278380 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278380 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3267dea6-9e40-494a-aeaa-458e00956620-config-out\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278380 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278380 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-config-volume\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278380 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278380 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94vk\" (UniqueName: \"kubernetes.io/projected/3267dea6-9e40-494a-aeaa-458e00956620-kube-api-access-x94vk\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278583 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3267dea6-9e40-494a-aeaa-458e00956620-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278583 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-web-config\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278583 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.278583 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.278471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379482 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379482 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3267dea6-9e40-494a-aeaa-458e00956620-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379704 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3267dea6-9e40-494a-aeaa-458e00956620-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379704 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3267dea6-9e40-494a-aeaa-458e00956620-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379809 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379809 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3267dea6-9e40-494a-aeaa-458e00956620-config-out\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379809 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379987 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-config-volume\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379987 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379987 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94vk\" (UniqueName: \"kubernetes.io/projected/3267dea6-9e40-494a-aeaa-458e00956620-kube-api-access-x94vk\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.379987 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3267dea6-9e40-494a-aeaa-458e00956620-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.380174 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.379998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-web-config\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.380174 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.380032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.380174 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.380070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3267dea6-9e40-494a-aeaa-458e00956620-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.380326 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.380234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3267dea6-9e40-494a-aeaa-458e00956620-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.381263 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.381233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3267dea6-9e40-494a-aeaa-458e00956620-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.382556 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.382531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.382663 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.382592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-config-volume\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.382800 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.382772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3267dea6-9e40-494a-aeaa-458e00956620-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.382944 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.382884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.383140 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.383120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.383309 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.383289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.383630 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.383613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3267dea6-9e40-494a-aeaa-458e00956620-config-out\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.384289 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.384271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-web-config\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.384702 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.384676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3267dea6-9e40-494a-aeaa-458e00956620-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.389482 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.389461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94vk\" (UniqueName: \"kubernetes.io/projected/3267dea6-9e40-494a-aeaa-458e00956620-kube-api-access-x94vk\") pod \"alertmanager-main-0\" (UID: \"3267dea6-9e40-494a-aeaa-458e00956620\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.495470 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.495446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:31:53.772239 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:53.772131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:31:53.774876 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:31:53.774850    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3267dea6_9e40_494a_aeaa_458e00956620.slice/crio-96e7f6d371350c2de0d49bd6799d9a5616e4dd7dbba15e4e7ac7a69c85752e62 WatchSource:0}: Error finding container 96e7f6d371350c2de0d49bd6799d9a5616e4dd7dbba15e4e7ac7a69c85752e62: Status 404 returned error can't find the container with id 96e7f6d371350c2de0d49bd6799d9a5616e4dd7dbba15e4e7ac7a69c85752e62
Sep 17 18:31:54.123450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:54.123362    2577 generic.go:356] "Generic (PLEG): container finished" podID="3267dea6-9e40-494a-aeaa-458e00956620" containerID="eae0a7d95fbc8ff77978e00ac746df3a48f04e803dad8934de519fa0e329b844" exitCode=0
Sep 17 18:31:54.123777 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:54.123452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3267dea6-9e40-494a-aeaa-458e00956620","Type":"ContainerDied","Data":"eae0a7d95fbc8ff77978e00ac746df3a48f04e803dad8934de519fa0e329b844"}
Sep 17 18:31:54.123777 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:54.123532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3267dea6-9e40-494a-aeaa-458e00956620","Type":"ContainerStarted","Data":"96e7f6d371350c2de0d49bd6799d9a5616e4dd7dbba15e4e7ac7a69c85752e62"}
Sep 17 18:31:55.130494 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:55.130452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3267dea6-9e40-494a-aeaa-458e00956620","Type":"ContainerStarted","Data":"1a4a50c91c244d4f69542d9c23275fcffe37e4d9236beefa795454a38dd2e360"}
Sep 17 18:31:55.130876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:55.130548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3267dea6-9e40-494a-aeaa-458e00956620","Type":"ContainerStarted","Data":"a1bbbf20f041e394c35f680a4d6e100c3e8256e51c93da84b420d91b73bfd2ca"}
Sep 17 18:31:55.130876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:55.130595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3267dea6-9e40-494a-aeaa-458e00956620","Type":"ContainerStarted","Data":"2b3130e644ac2ec7bd78dd0afb00ba0eb5f0e30e695080f91ed1e28b5c53d629"}
Sep 17 18:31:55.130876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:55.130633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3267dea6-9e40-494a-aeaa-458e00956620","Type":"ContainerStarted","Data":"44f0b0c75b248f6976bc42e0af94c2d3569efdf14ea1e9ae77f3abfc4e70b76d"}
Sep 17 18:31:55.130876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:55.130673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3267dea6-9e40-494a-aeaa-458e00956620","Type":"ContainerStarted","Data":"22954a4f4b841e2129cc265fcd8e7763a5b34f41cc960352fd12ce4a8ef65f9f"}
Sep 17 18:31:55.130876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:55.130711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3267dea6-9e40-494a-aeaa-458e00956620","Type":"ContainerStarted","Data":"90db75491d22db83194cb152e3eec7107eb8be40f358c66cdd0631722003c0a9"}
Sep 17 18:31:55.159866 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:31:55.159798    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.144449204 podStartE2EDuration="2.159777573s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.328369ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:55.156671954 +0000 UTC m=+150.203074881" watchObservedRunningTime="2026-09-17 18:31:55.159777573 +0000 UTC m=+150.206180476"
Sep 17 18:32:16.937747 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:16.937665    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66dd5c4f9b-rr7qm"]
Sep 17 18:32:16.943982 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:16.941283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:16.953506 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:16.953485    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66dd5c4f9b-rr7qm"]
Sep 17 18:32:17.075703 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.075666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-config\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.075871 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.075714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-oauth-serving-cert\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.075871 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.075739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-trusted-ca-bundle\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.075871 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.075754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb64d\" (UniqueName: \"kubernetes.io/projected/c717e2b1-1d25-4231-a367-e2398cbff6fd-kube-api-access-rb64d\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.075871 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.075779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-serving-cert\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.075871 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.075841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-oauth-config\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.076098 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.075890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-service-ca\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.176897 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.176860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-serving-cert\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.177063 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.176957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-oauth-config\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.177063 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.176988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-service-ca\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.177063 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.177024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-config\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.177171 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.177068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-oauth-serving-cert\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.177171 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.177100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-trusted-ca-bundle\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.177171 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.177124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb64d\" (UniqueName: \"kubernetes.io/projected/c717e2b1-1d25-4231-a367-e2398cbff6fd-kube-api-access-rb64d\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.177820 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.177792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-config\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.177950 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.177816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-service-ca\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.178028 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.178004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-oauth-serving-cert\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.178113 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.178099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-trusted-ca-bundle\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.179444 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.179424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-oauth-config\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.179546 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.179529    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-serving-cert\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.190422 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.190368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb64d\" (UniqueName: \"kubernetes.io/projected/c717e2b1-1d25-4231-a367-e2398cbff6fd-kube-api-access-rb64d\") pod \"console-66dd5c4f9b-rr7qm\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") " pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.251649 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.251627    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:17.523747 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:17.523720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66dd5c4f9b-rr7qm"]
Sep 17 18:32:17.526233 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:32:17.526202    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc717e2b1_1d25_4231_a367_e2398cbff6fd.slice/crio-3c236007e42f4e3e0f32802590c30f1b58eb4dbd2e2dd1761bde6ffc5aa5f3fb WatchSource:0}: Error finding container 3c236007e42f4e3e0f32802590c30f1b58eb4dbd2e2dd1761bde6ffc5aa5f3fb: Status 404 returned error can't find the container with id 3c236007e42f4e3e0f32802590c30f1b58eb4dbd2e2dd1761bde6ffc5aa5f3fb
Sep 17 18:32:18.224386 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:18.224347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66dd5c4f9b-rr7qm" event={"ID":"c717e2b1-1d25-4231-a367-e2398cbff6fd","Type":"ContainerStarted","Data":"02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851"}
Sep 17 18:32:18.224831 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:18.224395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66dd5c4f9b-rr7qm" event={"ID":"c717e2b1-1d25-4231-a367-e2398cbff6fd","Type":"ContainerStarted","Data":"3c236007e42f4e3e0f32802590c30f1b58eb4dbd2e2dd1761bde6ffc5aa5f3fb"}
Sep 17 18:32:18.247254 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:18.247208    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66dd5c4f9b-rr7qm" podStartSLOduration=2.24719438 podStartE2EDuration="2.24719438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:18.245000421 +0000 UTC m=+173.291403362" watchObservedRunningTime="2026-09-17 18:32:18.24719438 +0000 UTC m=+173.293597282"
Sep 17 18:32:27.252773 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:27.252735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:27.253170 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:27.252789    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:27.257384 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:27.257362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:27.266106 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:27.266087    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:32:27.321610 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:27.321571    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf956847b-fg2mj"]
Sep 17 18:32:52.341401 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.341337    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bf956847b-fg2mj" podUID="7c689fe4-4f75-4802-b2c9-830990295a35" containerName="console" containerID="cri-o://b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447" gracePeriod=15
Sep 17 18:32:52.573676 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.573653    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:32:52.756164 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756127    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-console-config\") pod \"7c689fe4-4f75-4802-b2c9-830990295a35\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") "
Sep 17 18:32:52.756352 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756172    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-oauth-config\") pod \"7c689fe4-4f75-4802-b2c9-830990295a35\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") "
Sep 17 18:32:52.756352 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756202    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljk5g\" (UniqueName: \"kubernetes.io/projected/7c689fe4-4f75-4802-b2c9-830990295a35-kube-api-access-ljk5g\") pod \"7c689fe4-4f75-4802-b2c9-830990295a35\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") "
Sep 17 18:32:52.756352 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756243    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-serving-cert\") pod \"7c689fe4-4f75-4802-b2c9-830990295a35\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") "
Sep 17 18:32:52.756352 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756279    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-trusted-ca-bundle\") pod \"7c689fe4-4f75-4802-b2c9-830990295a35\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") "
Sep 17 18:32:52.756352 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756321    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-oauth-serving-cert\") pod \"7c689fe4-4f75-4802-b2c9-830990295a35\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") "
Sep 17 18:32:52.756665 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756372    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-service-ca\") pod \"7c689fe4-4f75-4802-b2c9-830990295a35\" (UID: \"7c689fe4-4f75-4802-b2c9-830990295a35\") "
Sep 17 18:32:52.756665 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756558    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-console-config" (OuterVolumeSpecName: "console-config") pod "7c689fe4-4f75-4802-b2c9-830990295a35" (UID: "7c689fe4-4f75-4802-b2c9-830990295a35"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:52.756795 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756764    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7c689fe4-4f75-4802-b2c9-830990295a35" (UID: "7c689fe4-4f75-4802-b2c9-830990295a35"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:52.756795 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756789    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-console-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:32:52.756898 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756804    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7c689fe4-4f75-4802-b2c9-830990295a35" (UID: "7c689fe4-4f75-4802-b2c9-830990295a35"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:52.756898 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.756878    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-service-ca" (OuterVolumeSpecName: "service-ca") pod "7c689fe4-4f75-4802-b2c9-830990295a35" (UID: "7c689fe4-4f75-4802-b2c9-830990295a35"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:52.758393 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.758370    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c689fe4-4f75-4802-b2c9-830990295a35-kube-api-access-ljk5g" (OuterVolumeSpecName: "kube-api-access-ljk5g") pod "7c689fe4-4f75-4802-b2c9-830990295a35" (UID: "7c689fe4-4f75-4802-b2c9-830990295a35"). InnerVolumeSpecName "kube-api-access-ljk5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:52.758789 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.758765    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7c689fe4-4f75-4802-b2c9-830990295a35" (UID: "7c689fe4-4f75-4802-b2c9-830990295a35"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:52.758789 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.758782    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7c689fe4-4f75-4802-b2c9-830990295a35" (UID: "7c689fe4-4f75-4802-b2c9-830990295a35"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:52.857973 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.857928    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-trusted-ca-bundle\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:32:52.858130 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.857988    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-oauth-serving-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:32:52.858130 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.858000    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c689fe4-4f75-4802-b2c9-830990295a35-service-ca\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:32:52.858130 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.858009    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-oauth-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:32:52.858130 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.858018    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljk5g\" (UniqueName: \"kubernetes.io/projected/7c689fe4-4f75-4802-b2c9-830990295a35-kube-api-access-ljk5g\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:32:52.858130 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:52.858027    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c689fe4-4f75-4802-b2c9-830990295a35-console-serving-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:32:53.370369 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:53.370337    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c689fe4-4f75-4802-b2c9-830990295a35" containerID="b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447" exitCode=0
Sep 17 18:32:53.370821 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:53.370382    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf956847b-fg2mj"
Sep 17 18:32:53.370821 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:53.370413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf956847b-fg2mj" event={"ID":"7c689fe4-4f75-4802-b2c9-830990295a35","Type":"ContainerDied","Data":"b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447"}
Sep 17 18:32:53.370821 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:53.370461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf956847b-fg2mj" event={"ID":"7c689fe4-4f75-4802-b2c9-830990295a35","Type":"ContainerDied","Data":"f9d365df704a43a2b8edb9509a031cfdb21ec76476080aee9f8a4193928a445f"}
Sep 17 18:32:53.370821 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:53.370483    2577 scope.go:117] "RemoveContainer" containerID="b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447"
Sep 17 18:32:53.378401 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:53.378383    2577 scope.go:117] "RemoveContainer" containerID="b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447"
Sep 17 18:32:53.378675 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:32:53.378658    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447\": container with ID starting with b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447 not found: ID does not exist" containerID="b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447"
Sep 17 18:32:53.378743 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:53.378686    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447"} err="failed to get container status \"b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447\": rpc error: code = NotFound desc = could not find container \"b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447\": container with ID starting with b68e90fd5a8fce5a2172ed84043df4733de0e5dba5f30a56c258b172c8353447 not found: ID does not exist"
Sep 17 18:32:53.386470 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:53.386447    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf956847b-fg2mj"]
Sep 17 18:32:53.390370 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:53.390349    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bf956847b-fg2mj"]
Sep 17 18:32:55.237914 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:32:55.237880    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c689fe4-4f75-4802-b2c9-830990295a35" path="/var/lib/kubelet/pods/7c689fe4-4f75-4802-b2c9-830990295a35/volumes"
Sep 17 18:33:05.999450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:05.999414    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"]
Sep 17 18:33:05.999854 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:05.999738    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c689fe4-4f75-4802-b2c9-830990295a35" containerName="console"
Sep 17 18:33:05.999854 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:05.999749    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c689fe4-4f75-4802-b2c9-830990295a35" containerName="console"
Sep 17 18:33:05.999854 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:05.999813    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c689fe4-4f75-4802-b2c9-830990295a35" containerName="console"
Sep 17 18:33:06.002708 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.002686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.005202 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.005183    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-69wft\""
Sep 17 18:33:06.005297 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.005279    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:33:06.006220 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.006195    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:33:06.011039 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.011012    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"]
Sep 17 18:33:06.060505 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.060462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlf2q\" (UniqueName: \"kubernetes.io/projected/9074d7c3-f599-4550-bf56-15d34b733096-kube-api-access-vlf2q\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.060715 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.060515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.060715 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.060651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.161480 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.161443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.161634 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.161513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.161634 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.161550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlf2q\" (UniqueName: \"kubernetes.io/projected/9074d7c3-f599-4550-bf56-15d34b733096-kube-api-access-vlf2q\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.161969 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.161947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.162047 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.161984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.170882 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.170848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlf2q\" (UniqueName: \"kubernetes.io/projected/9074d7c3-f599-4550-bf56-15d34b733096-kube-api-access-vlf2q\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.312433 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.312337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:06.584512 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:06.584488    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"]
Sep 17 18:33:06.586803 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:33:06.586776    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9074d7c3_f599_4550_bf56_15d34b733096.slice/crio-6b35ecac9f0cd65c487ae0886845332bb9e4822430f8f0241ee289c5af75a8ae WatchSource:0}: Error finding container 6b35ecac9f0cd65c487ae0886845332bb9e4822430f8f0241ee289c5af75a8ae: Status 404 returned error can't find the container with id 6b35ecac9f0cd65c487ae0886845332bb9e4822430f8f0241ee289c5af75a8ae
Sep 17 18:33:07.428297 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:07.428253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d" event={"ID":"9074d7c3-f599-4550-bf56-15d34b733096","Type":"ContainerStarted","Data":"6b35ecac9f0cd65c487ae0886845332bb9e4822430f8f0241ee289c5af75a8ae"}
Sep 17 18:33:13.457260 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:13.457223    2577 generic.go:356] "Generic (PLEG): container finished" podID="9074d7c3-f599-4550-bf56-15d34b733096" containerID="e7ad887943706c28fb43e834d509197225b035f20f3bd20aced51f3e7421cd13" exitCode=0
Sep 17 18:33:13.457641 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:13.457320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d" event={"ID":"9074d7c3-f599-4550-bf56-15d34b733096","Type":"ContainerDied","Data":"e7ad887943706c28fb43e834d509197225b035f20f3bd20aced51f3e7421cd13"}
Sep 17 18:33:20.489286 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:20.489248    2577 generic.go:356] "Generic (PLEG): container finished" podID="9074d7c3-f599-4550-bf56-15d34b733096" containerID="833891d58e5743e7a920f7f9d19ac3da8aee28f504bc70dba92f08a67009a53f" exitCode=0
Sep 17 18:33:20.489680 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:20.489327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d" event={"ID":"9074d7c3-f599-4550-bf56-15d34b733096","Type":"ContainerDied","Data":"833891d58e5743e7a920f7f9d19ac3da8aee28f504bc70dba92f08a67009a53f"}
Sep 17 18:33:28.527713 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:28.527674    2577 generic.go:356] "Generic (PLEG): container finished" podID="9074d7c3-f599-4550-bf56-15d34b733096" containerID="f258184fd2af8a911152a4faa9467a3935f0d0f98c417fd216d503b325c009cc" exitCode=0
Sep 17 18:33:28.528089 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:28.527761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d" event={"ID":"9074d7c3-f599-4550-bf56-15d34b733096","Type":"ContainerDied","Data":"f258184fd2af8a911152a4faa9467a3935f0d0f98c417fd216d503b325c009cc"}
Sep 17 18:33:29.660846 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:29.660824    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:29.773509 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:29.773476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlf2q\" (UniqueName: \"kubernetes.io/projected/9074d7c3-f599-4550-bf56-15d34b733096-kube-api-access-vlf2q\") pod \"9074d7c3-f599-4550-bf56-15d34b733096\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") "
Sep 17 18:33:29.773675 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:29.773529    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-util\") pod \"9074d7c3-f599-4550-bf56-15d34b733096\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") "
Sep 17 18:33:29.773675 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:29.773596    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-bundle\") pod \"9074d7c3-f599-4550-bf56-15d34b733096\" (UID: \"9074d7c3-f599-4550-bf56-15d34b733096\") "
Sep 17 18:33:29.774331 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:29.774298    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-bundle" (OuterVolumeSpecName: "bundle") pod "9074d7c3-f599-4550-bf56-15d34b733096" (UID: "9074d7c3-f599-4550-bf56-15d34b733096"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:29.775766 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:29.775743    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9074d7c3-f599-4550-bf56-15d34b733096-kube-api-access-vlf2q" (OuterVolumeSpecName: "kube-api-access-vlf2q") pod "9074d7c3-f599-4550-bf56-15d34b733096" (UID: "9074d7c3-f599-4550-bf56-15d34b733096"). InnerVolumeSpecName "kube-api-access-vlf2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:29.777975 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:29.777954    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-util" (OuterVolumeSpecName: "util") pod "9074d7c3-f599-4550-bf56-15d34b733096" (UID: "9074d7c3-f599-4550-bf56-15d34b733096"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:29.874377 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:29.874294    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlf2q\" (UniqueName: \"kubernetes.io/projected/9074d7c3-f599-4550-bf56-15d34b733096-kube-api-access-vlf2q\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:33:29.874377 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:29.874328    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-util\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:33:29.874377 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:29.874337    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9074d7c3-f599-4550-bf56-15d34b733096-bundle\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:33:30.537787 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:30.537758    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d"
Sep 17 18:33:30.537787 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:30.537775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awnt9d" event={"ID":"9074d7c3-f599-4550-bf56-15d34b733096","Type":"ContainerDied","Data":"6b35ecac9f0cd65c487ae0886845332bb9e4822430f8f0241ee289c5af75a8ae"}
Sep 17 18:33:30.538017 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:30.537817    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b35ecac9f0cd65c487ae0886845332bb9e4822430f8f0241ee289c5af75a8ae"
Sep 17 18:33:35.052881 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.052841    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"]
Sep 17 18:33:35.053319 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.053196    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9074d7c3-f599-4550-bf56-15d34b733096" containerName="pull"
Sep 17 18:33:35.053319 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.053209    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9074d7c3-f599-4550-bf56-15d34b733096" containerName="pull"
Sep 17 18:33:35.053319 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.053217    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9074d7c3-f599-4550-bf56-15d34b733096" containerName="util"
Sep 17 18:33:35.053319 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.053222    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9074d7c3-f599-4550-bf56-15d34b733096" containerName="util"
Sep 17 18:33:35.053319 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.053228    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9074d7c3-f599-4550-bf56-15d34b733096" containerName="extract"
Sep 17 18:33:35.053319 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.053234    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9074d7c3-f599-4550-bf56-15d34b733096" containerName="extract"
Sep 17 18:33:35.053319 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.053291    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9074d7c3-f599-4550-bf56-15d34b733096" containerName="extract"
Sep 17 18:33:35.057045 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.057027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"
Sep 17 18:33:35.064657 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.064636    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-v525v\""
Sep 17 18:33:35.064920 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.064888    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 18:33:35.065342 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.065325    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 18:33:35.065389 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.065328    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 18:33:35.076726 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.076700    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"]
Sep 17 18:33:35.220257 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.220213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5tb\" (UniqueName: \"kubernetes.io/projected/1cd04648-4b0c-47fc-bd85-c0cd8b65734a-kube-api-access-hb5tb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-whqcn\" (UID: \"1cd04648-4b0c-47fc-bd85-c0cd8b65734a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"
Sep 17 18:33:35.220518 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.220268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1cd04648-4b0c-47fc-bd85-c0cd8b65734a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-whqcn\" (UID: \"1cd04648-4b0c-47fc-bd85-c0cd8b65734a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"
Sep 17 18:33:35.321530 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.321447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1cd04648-4b0c-47fc-bd85-c0cd8b65734a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-whqcn\" (UID: \"1cd04648-4b0c-47fc-bd85-c0cd8b65734a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"
Sep 17 18:33:35.321677 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.321538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5tb\" (UniqueName: \"kubernetes.io/projected/1cd04648-4b0c-47fc-bd85-c0cd8b65734a-kube-api-access-hb5tb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-whqcn\" (UID: \"1cd04648-4b0c-47fc-bd85-c0cd8b65734a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"
Sep 17 18:33:35.323839 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.323807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1cd04648-4b0c-47fc-bd85-c0cd8b65734a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-whqcn\" (UID: \"1cd04648-4b0c-47fc-bd85-c0cd8b65734a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"
Sep 17 18:33:35.348635 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.348607    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5tb\" (UniqueName: \"kubernetes.io/projected/1cd04648-4b0c-47fc-bd85-c0cd8b65734a-kube-api-access-hb5tb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-whqcn\" (UID: \"1cd04648-4b0c-47fc-bd85-c0cd8b65734a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"
Sep 17 18:33:35.366603 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.366565    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"
Sep 17 18:33:35.643553 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:35.643510    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"]
Sep 17 18:33:35.646105 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:33:35.646077    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd04648_4b0c_47fc_bd85_c0cd8b65734a.slice/crio-99c84d9d67981b7616f8f6bcbf4fc8025e7294a85b54094d38abed5bcaf9eb87 WatchSource:0}: Error finding container 99c84d9d67981b7616f8f6bcbf4fc8025e7294a85b54094d38abed5bcaf9eb87: Status 404 returned error can't find the container with id 99c84d9d67981b7616f8f6bcbf4fc8025e7294a85b54094d38abed5bcaf9eb87
Sep 17 18:33:36.563124 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:36.563080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn" event={"ID":"1cd04648-4b0c-47fc-bd85-c0cd8b65734a","Type":"ContainerStarted","Data":"99c84d9d67981b7616f8f6bcbf4fc8025e7294a85b54094d38abed5bcaf9eb87"}
Sep 17 18:33:38.574472 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:38.574430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn" event={"ID":"1cd04648-4b0c-47fc-bd85-c0cd8b65734a","Type":"ContainerStarted","Data":"2a94e4ec5055f85606fc4120d0be2d228f5dbf8572c9393831aabc74d0947062"}
Sep 17 18:33:38.574991 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:38.574542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"
Sep 17 18:33:38.603295 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:38.603227    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn" podStartSLOduration=0.805198289 podStartE2EDuration="3.603203892s" totalImagesPullingTime="2.798005603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:38.600501573 +0000 UTC m=+253.646904491" watchObservedRunningTime="2026-09-17 18:33:38.603203892 +0000 UTC m=+253.649606795"
Sep 17 18:33:39.173657 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.173611    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6v6z2"]
Sep 17 18:33:39.177047 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.177024    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.181666 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.181645    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 18:33:39.181666 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.181645    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-l8pgf\""
Sep 17 18:33:39.181824 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.181702    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 18:33:39.187674 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.187652    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6v6z2"]
Sep 17 18:33:39.357190 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.357159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/38a3400e-cde5-4535-b258-bba2bf45ede6-cabundle0\") pod \"keda-operator-579c8958f7-6v6z2\" (UID: \"38a3400e-cde5-4535-b258-bba2bf45ede6\") " pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.357369 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.357195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4sj6\" (UniqueName: \"kubernetes.io/projected/38a3400e-cde5-4535-b258-bba2bf45ede6-kube-api-access-r4sj6\") pod \"keda-operator-579c8958f7-6v6z2\" (UID: \"38a3400e-cde5-4535-b258-bba2bf45ede6\") " pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.357369 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.357258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/38a3400e-cde5-4535-b258-bba2bf45ede6-certificates\") pod \"keda-operator-579c8958f7-6v6z2\" (UID: \"38a3400e-cde5-4535-b258-bba2bf45ede6\") " pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.398398 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.398360    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"]
Sep 17 18:33:39.401642 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.401625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:39.404732 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.404708    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 18:33:39.415517 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.415493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"]
Sep 17 18:33:39.458482 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.458398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/38a3400e-cde5-4535-b258-bba2bf45ede6-cabundle0\") pod \"keda-operator-579c8958f7-6v6z2\" (UID: \"38a3400e-cde5-4535-b258-bba2bf45ede6\") " pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.458482 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.458433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4sj6\" (UniqueName: \"kubernetes.io/projected/38a3400e-cde5-4535-b258-bba2bf45ede6-kube-api-access-r4sj6\") pod \"keda-operator-579c8958f7-6v6z2\" (UID: \"38a3400e-cde5-4535-b258-bba2bf45ede6\") " pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.458704 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.458581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/38a3400e-cde5-4535-b258-bba2bf45ede6-certificates\") pod \"keda-operator-579c8958f7-6v6z2\" (UID: \"38a3400e-cde5-4535-b258-bba2bf45ede6\") " pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.458784 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:33:39.458767    2577 secret.go:281] references non-existent secret key: ca.crt
Sep 17 18:33:39.458842 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:33:39.458786    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 18:33:39.458842 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:33:39.458794    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-6v6z2: references non-existent secret key: ca.crt
Sep 17 18:33:39.458977 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:33:39.458846    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38a3400e-cde5-4535-b258-bba2bf45ede6-certificates podName:38a3400e-cde5-4535-b258-bba2bf45ede6 nodeName:}" failed. No retries permitted until 2026-09-17 18:33:39.958828779 +0000 UTC m=+255.005231672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/38a3400e-cde5-4535-b258-bba2bf45ede6-certificates") pod "keda-operator-579c8958f7-6v6z2" (UID: "38a3400e-cde5-4535-b258-bba2bf45ede6") : references non-existent secret key: ca.crt
Sep 17 18:33:39.459184 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.459163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/38a3400e-cde5-4535-b258-bba2bf45ede6-cabundle0\") pod \"keda-operator-579c8958f7-6v6z2\" (UID: \"38a3400e-cde5-4535-b258-bba2bf45ede6\") " pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.469749 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.469722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4sj6\" (UniqueName: \"kubernetes.io/projected/38a3400e-cde5-4535-b258-bba2bf45ede6-kube-api-access-r4sj6\") pod \"keda-operator-579c8958f7-6v6z2\" (UID: \"38a3400e-cde5-4535-b258-bba2bf45ede6\") " pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.560005 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.559953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/23829eda-5340-4742-a0f2-9ae178e8a5cc-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rjjbh\" (UID: \"23829eda-5340-4742-a0f2-9ae178e8a5cc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:39.560201 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.560064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/23829eda-5340-4742-a0f2-9ae178e8a5cc-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rjjbh\" (UID: \"23829eda-5340-4742-a0f2-9ae178e8a5cc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:39.560201 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.560140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tbbx\" (UniqueName: \"kubernetes.io/projected/23829eda-5340-4742-a0f2-9ae178e8a5cc-kube-api-access-6tbbx\") pod \"keda-metrics-apiserver-77857b7b98-rjjbh\" (UID: \"23829eda-5340-4742-a0f2-9ae178e8a5cc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:39.661091 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.661038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/23829eda-5340-4742-a0f2-9ae178e8a5cc-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rjjbh\" (UID: \"23829eda-5340-4742-a0f2-9ae178e8a5cc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:39.661488 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.661125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/23829eda-5340-4742-a0f2-9ae178e8a5cc-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rjjbh\" (UID: \"23829eda-5340-4742-a0f2-9ae178e8a5cc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:39.661488 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.661173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tbbx\" (UniqueName: \"kubernetes.io/projected/23829eda-5340-4742-a0f2-9ae178e8a5cc-kube-api-access-6tbbx\") pod \"keda-metrics-apiserver-77857b7b98-rjjbh\" (UID: \"23829eda-5340-4742-a0f2-9ae178e8a5cc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:39.661488 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:33:39.661276    2577 secret.go:281] references non-existent secret key: tls.crt
Sep 17 18:33:39.661488 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:33:39.661293    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 18:33:39.661488 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:33:39.661311    2577 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 17 18:33:39.661488 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:33:39.661335    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 17 18:33:39.661488 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:33:39.661388    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23829eda-5340-4742-a0f2-9ae178e8a5cc-certificates podName:23829eda-5340-4742-a0f2-9ae178e8a5cc nodeName:}" failed. No retries permitted until 2026-09-17 18:33:40.161369961 +0000 UTC m=+255.207772846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/23829eda-5340-4742-a0f2-9ae178e8a5cc-certificates") pod "keda-metrics-apiserver-77857b7b98-rjjbh" (UID: "23829eda-5340-4742-a0f2-9ae178e8a5cc") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 17 18:33:39.661488 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.661412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/23829eda-5340-4742-a0f2-9ae178e8a5cc-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rjjbh\" (UID: \"23829eda-5340-4742-a0f2-9ae178e8a5cc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:39.670403 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.670375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tbbx\" (UniqueName: \"kubernetes.io/projected/23829eda-5340-4742-a0f2-9ae178e8a5cc-kube-api-access-6tbbx\") pod \"keda-metrics-apiserver-77857b7b98-rjjbh\" (UID: \"23829eda-5340-4742-a0f2-9ae178e8a5cc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:39.963640 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.963605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/38a3400e-cde5-4535-b258-bba2bf45ede6-certificates\") pod \"keda-operator-579c8958f7-6v6z2\" (UID: \"38a3400e-cde5-4535-b258-bba2bf45ede6\") " pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.966205 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.966181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/38a3400e-cde5-4535-b258-bba2bf45ede6-certificates\") pod \"keda-operator-579c8958f7-6v6z2\" (UID: \"38a3400e-cde5-4535-b258-bba2bf45ede6\") " pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:39.997320 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:39.997289    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-s8nb5"]
Sep 17 18:33:40.000833 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.000817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-s8nb5"
Sep 17 18:33:40.004129 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.004106    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 18:33:40.023894 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.023868    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-s8nb5"]
Sep 17 18:33:40.087255 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.087220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:40.165480 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.165446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvs7\" (UniqueName: \"kubernetes.io/projected/cd0e2a95-8f27-42bf-a104-7d58f49f22cc-kube-api-access-ljvs7\") pod \"keda-admission-6c4d864b86-s8nb5\" (UID: \"cd0e2a95-8f27-42bf-a104-7d58f49f22cc\") " pod="openshift-keda/keda-admission-6c4d864b86-s8nb5"
Sep 17 18:33:40.165654 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.165521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/23829eda-5340-4742-a0f2-9ae178e8a5cc-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rjjbh\" (UID: \"23829eda-5340-4742-a0f2-9ae178e8a5cc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:40.165654 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.165544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd0e2a95-8f27-42bf-a104-7d58f49f22cc-certificates\") pod \"keda-admission-6c4d864b86-s8nb5\" (UID: \"cd0e2a95-8f27-42bf-a104-7d58f49f22cc\") " pod="openshift-keda/keda-admission-6c4d864b86-s8nb5"
Sep 17 18:33:40.168189 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.168136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/23829eda-5340-4742-a0f2-9ae178e8a5cc-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rjjbh\" (UID: \"23829eda-5340-4742-a0f2-9ae178e8a5cc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:40.266452 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.266358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvs7\" (UniqueName: \"kubernetes.io/projected/cd0e2a95-8f27-42bf-a104-7d58f49f22cc-kube-api-access-ljvs7\") pod \"keda-admission-6c4d864b86-s8nb5\" (UID: \"cd0e2a95-8f27-42bf-a104-7d58f49f22cc\") " pod="openshift-keda/keda-admission-6c4d864b86-s8nb5"
Sep 17 18:33:40.266630 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.266455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd0e2a95-8f27-42bf-a104-7d58f49f22cc-certificates\") pod \"keda-admission-6c4d864b86-s8nb5\" (UID: \"cd0e2a95-8f27-42bf-a104-7d58f49f22cc\") " pod="openshift-keda/keda-admission-6c4d864b86-s8nb5"
Sep 17 18:33:40.268961 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.268935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd0e2a95-8f27-42bf-a104-7d58f49f22cc-certificates\") pod \"keda-admission-6c4d864b86-s8nb5\" (UID: \"cd0e2a95-8f27-42bf-a104-7d58f49f22cc\") " pod="openshift-keda/keda-admission-6c4d864b86-s8nb5"
Sep 17 18:33:40.280466 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.280433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvs7\" (UniqueName: \"kubernetes.io/projected/cd0e2a95-8f27-42bf-a104-7d58f49f22cc-kube-api-access-ljvs7\") pod \"keda-admission-6c4d864b86-s8nb5\" (UID: \"cd0e2a95-8f27-42bf-a104-7d58f49f22cc\") " pod="openshift-keda/keda-admission-6c4d864b86-s8nb5"
Sep 17 18:33:40.312304 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.312260    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-s8nb5"
Sep 17 18:33:40.312475 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.312407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:40.365561 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.365511    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6v6z2"]
Sep 17 18:33:40.371543 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:33:40.371271    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a3400e_cde5_4535_b258_bba2bf45ede6.slice/crio-9e2304fd393582eed03973cc1ccbe204783126fa9adb440ff7396e84bf7a761f WatchSource:0}: Error finding container 9e2304fd393582eed03973cc1ccbe204783126fa9adb440ff7396e84bf7a761f: Status 404 returned error can't find the container with id 9e2304fd393582eed03973cc1ccbe204783126fa9adb440ff7396e84bf7a761f
Sep 17 18:33:40.584749 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.584678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-6v6z2" event={"ID":"38a3400e-cde5-4535-b258-bba2bf45ede6","Type":"ContainerStarted","Data":"9e2304fd393582eed03973cc1ccbe204783126fa9adb440ff7396e84bf7a761f"}
Sep 17 18:33:40.602283 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.602250    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-s8nb5"]
Sep 17 18:33:40.604405 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:40.604381    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"]
Sep 17 18:33:40.604535 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:33:40.604418    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0e2a95_8f27_42bf_a104_7d58f49f22cc.slice/crio-8ad7b339a5e363666e788e133286204c29bc04e37267d8c434946222ad58e367 WatchSource:0}: Error finding container 8ad7b339a5e363666e788e133286204c29bc04e37267d8c434946222ad58e367: Status 404 returned error can't find the container with id 8ad7b339a5e363666e788e133286204c29bc04e37267d8c434946222ad58e367
Sep 17 18:33:40.610665 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:33:40.610639    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23829eda_5340_4742_a0f2_9ae178e8a5cc.slice/crio-9a3fbba15210f2d1ffa95ab42a46576f7ce1a5e43954db55cd9bae65eb51a24f WatchSource:0}: Error finding container 9a3fbba15210f2d1ffa95ab42a46576f7ce1a5e43954db55cd9bae65eb51a24f: Status 404 returned error can't find the container with id 9a3fbba15210f2d1ffa95ab42a46576f7ce1a5e43954db55cd9bae65eb51a24f
Sep 17 18:33:41.592524 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:41.592479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh" event={"ID":"23829eda-5340-4742-a0f2-9ae178e8a5cc","Type":"ContainerStarted","Data":"9a3fbba15210f2d1ffa95ab42a46576f7ce1a5e43954db55cd9bae65eb51a24f"}
Sep 17 18:33:41.594193 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:41.594156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-s8nb5" event={"ID":"cd0e2a95-8f27-42bf-a104-7d58f49f22cc","Type":"ContainerStarted","Data":"8ad7b339a5e363666e788e133286204c29bc04e37267d8c434946222ad58e367"}
Sep 17 18:33:45.617943 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:45.617886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh" event={"ID":"23829eda-5340-4742-a0f2-9ae178e8a5cc","Type":"ContainerStarted","Data":"8b7f183b19c8d2526832e2d9c1354d0b1c8a855a7800ea82a28acec24e6354c8"}
Sep 17 18:33:45.618383 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:45.618006    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:45.619667 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:45.619636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-6v6z2" event={"ID":"38a3400e-cde5-4535-b258-bba2bf45ede6","Type":"ContainerStarted","Data":"a4e6160d07fbaf75965833ba51f7bb20d519ce5120eeb6d67d90a2a4636d80fc"}
Sep 17 18:33:45.619813 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:45.619707    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:33:45.621251 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:45.621206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-s8nb5" event={"ID":"cd0e2a95-8f27-42bf-a104-7d58f49f22cc","Type":"ContainerStarted","Data":"14ddc2ec29da8cea20d1970685ddf932f6119b42d5fad0722ad123df0fef48e2"}
Sep 17 18:33:45.621355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:45.621346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-s8nb5"
Sep 17 18:33:45.639035 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:45.638994    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh" podStartSLOduration=2.547752595 podStartE2EDuration="6.638976528s" totalImagesPullingTime="4.091223933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:45.636577088 +0000 UTC m=+260.682980016" watchObservedRunningTime="2026-09-17 18:33:45.638976528 +0000 UTC m=+260.685379433"
Sep 17 18:33:45.653269 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:45.653215    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-s8nb5" podStartSLOduration=2.566208644 podStartE2EDuration="6.653199485s" totalImagesPullingTime="4.086990841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:45.651933037 +0000 UTC m=+260.698335963" watchObservedRunningTime="2026-09-17 18:33:45.653199485 +0000 UTC m=+260.699602388"
Sep 17 18:33:56.636447 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:56.636406    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rjjbh"
Sep 17 18:33:56.659752 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:56.659690    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-6v6z2" podStartSLOduration=13.327403834 podStartE2EDuration="17.659669448s" totalImagesPullingTime="4.332265614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:45.666140143 +0000 UTC m=+260.712543071" watchObservedRunningTime="2026-09-17 18:33:56.659669448 +0000 UTC m=+271.706072355"
Sep 17 18:33:59.581624 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:33:59.581589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-whqcn"
Sep 17 18:34:06.634320 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:06.634292    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-s8nb5"
Sep 17 18:34:06.634707 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:06.634360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-6v6z2"
Sep 17 18:34:27.118861 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:27.118833    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:34:57.334433 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.334403    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"]
Sep 17 18:34:57.336553 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.336535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:34:57.339214 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.339196    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:34:57.339981 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.339966    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 18:34:57.340055 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.340009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-vcbfl\""
Sep 17 18:34:57.340100 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.340063    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:34:57.355331 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.355309    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"]
Sep 17 18:34:57.458213 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.458174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4d2\" (UniqueName: \"kubernetes.io/projected/f8c1d308-5cbe-4e05-ae02-92f1721a0ede-kube-api-access-rh4d2\") pod \"llmisvc-controller-manager-645f9c8fd6-7pz7z\" (UID: \"f8c1d308-5cbe-4e05-ae02-92f1721a0ede\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:34:57.458391 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.458253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c1d308-5cbe-4e05-ae02-92f1721a0ede-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-7pz7z\" (UID: \"f8c1d308-5cbe-4e05-ae02-92f1721a0ede\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:34:57.559241 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.559204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4d2\" (UniqueName: \"kubernetes.io/projected/f8c1d308-5cbe-4e05-ae02-92f1721a0ede-kube-api-access-rh4d2\") pod \"llmisvc-controller-manager-645f9c8fd6-7pz7z\" (UID: \"f8c1d308-5cbe-4e05-ae02-92f1721a0ede\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:34:57.559401 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.559265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c1d308-5cbe-4e05-ae02-92f1721a0ede-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-7pz7z\" (UID: \"f8c1d308-5cbe-4e05-ae02-92f1721a0ede\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:34:57.559401 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:34:57.559365    2577 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 18:34:57.559474 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:34:57.559416    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8c1d308-5cbe-4e05-ae02-92f1721a0ede-cert podName:f8c1d308-5cbe-4e05-ae02-92f1721a0ede nodeName:}" failed. No retries permitted until 2026-09-17 18:34:58.059401145 +0000 UTC m=+333.105804026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f8c1d308-5cbe-4e05-ae02-92f1721a0ede-cert") pod "llmisvc-controller-manager-645f9c8fd6-7pz7z" (UID: "f8c1d308-5cbe-4e05-ae02-92f1721a0ede") : secret "llmisvc-webhook-server-cert" not found
Sep 17 18:34:57.569689 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:57.569659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4d2\" (UniqueName: \"kubernetes.io/projected/f8c1d308-5cbe-4e05-ae02-92f1721a0ede-kube-api-access-rh4d2\") pod \"llmisvc-controller-manager-645f9c8fd6-7pz7z\" (UID: \"f8c1d308-5cbe-4e05-ae02-92f1721a0ede\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:34:58.064458 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:58.064421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c1d308-5cbe-4e05-ae02-92f1721a0ede-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-7pz7z\" (UID: \"f8c1d308-5cbe-4e05-ae02-92f1721a0ede\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:34:58.066831 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:58.066810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c1d308-5cbe-4e05-ae02-92f1721a0ede-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-7pz7z\" (UID: \"f8c1d308-5cbe-4e05-ae02-92f1721a0ede\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:34:58.246157 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:58.246125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:34:58.517226 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:58.517200    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"]
Sep 17 18:34:58.519521 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:34:58.519493    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf8c1d308_5cbe_4e05_ae02_92f1721a0ede.slice/crio-3a70bb296e865bf6341dff10a49261d46d562446b8bb31864c652624ea4f69a3 WatchSource:0}: Error finding container 3a70bb296e865bf6341dff10a49261d46d562446b8bb31864c652624ea4f69a3: Status 404 returned error can't find the container with id 3a70bb296e865bf6341dff10a49261d46d562446b8bb31864c652624ea4f69a3
Sep 17 18:34:58.520618 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:58.520602    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:34:58.950886 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:34:58.950845    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z" event={"ID":"f8c1d308-5cbe-4e05-ae02-92f1721a0ede","Type":"ContainerStarted","Data":"3a70bb296e865bf6341dff10a49261d46d562446b8bb31864c652624ea4f69a3"}
Sep 17 18:35:01.973185 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:01.973138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z" event={"ID":"f8c1d308-5cbe-4e05-ae02-92f1721a0ede","Type":"ContainerStarted","Data":"4d24310592b45e21c5cb1e07285fbeda07b296850d4b637f705707bc2324c8cd"}
Sep 17 18:35:01.973671 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:01.973204    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:35:01.994354 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:01.994296    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z" podStartSLOduration=2.492112307 podStartE2EDuration="4.994274393s" totalImagesPullingTime="2.502162086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:01.99211406 +0000 UTC m=+337.038516962" watchObservedRunningTime="2026-09-17 18:35:01.994274393 +0000 UTC m=+337.040677295"
Sep 17 18:35:32.980405 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:32.980319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7pz7z"
Sep 17 18:35:34.700168 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.700131    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56d5947dc-ft2cl"]
Sep 17 18:35:34.721300 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.721266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.722497 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.722474    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d5947dc-ft2cl"]
Sep 17 18:35:34.794129 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.794096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-oauth-serving-cert\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.794337 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.794142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-console-config\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.794337 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.794175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-console-oauth-config\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.794337 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.794237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-service-ca\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.794337 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.794271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-trusted-ca-bundle\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.794337 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.794300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-console-serving-cert\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.794523 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.794345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw5br\" (UniqueName: \"kubernetes.io/projected/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-kube-api-access-cw5br\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.895125 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.895097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-service-ca\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.895125 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.895131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-trusted-ca-bundle\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.895337 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.895149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-console-serving-cert\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.895337 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.895165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw5br\" (UniqueName: \"kubernetes.io/projected/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-kube-api-access-cw5br\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.895337 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.895239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-oauth-serving-cert\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.895337 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.895267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-console-config\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.895337 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.895298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-console-oauth-config\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.895950 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.895925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-service-ca\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.896099 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.896036    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-oauth-serving-cert\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.896099 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.896045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-console-config\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.896313 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.896283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-trusted-ca-bundle\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.897707 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.897683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-console-serving-cert\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.897707 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.897700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-console-oauth-config\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:34.906299 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:34.906278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw5br\" (UniqueName: \"kubernetes.io/projected/81e08d30-3533-46b2-bdd4-b0a29d47ce1a-kube-api-access-cw5br\") pod \"console-56d5947dc-ft2cl\" (UID: \"81e08d30-3533-46b2-bdd4-b0a29d47ce1a\") " pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:35.031687 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:35.031603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:35.334583 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:35.334558    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d5947dc-ft2cl"]
Sep 17 18:35:35.336745 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:35:35.336720    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e08d30_3533_46b2_bdd4_b0a29d47ce1a.slice/crio-7d92941ae8e3465df2cd6c6c994328d5d907511c7ca9a862a616ee1baec55631 WatchSource:0}: Error finding container 7d92941ae8e3465df2cd6c6c994328d5d907511c7ca9a862a616ee1baec55631: Status 404 returned error can't find the container with id 7d92941ae8e3465df2cd6c6c994328d5d907511c7ca9a862a616ee1baec55631
Sep 17 18:35:36.124218 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:36.124183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d5947dc-ft2cl" event={"ID":"81e08d30-3533-46b2-bdd4-b0a29d47ce1a","Type":"ContainerStarted","Data":"f4d06bf45a6c32460c9bd5481ff3045ed033e19743aade05a36c276a710411c9"}
Sep 17 18:35:36.124623 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:36.124232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d5947dc-ft2cl" event={"ID":"81e08d30-3533-46b2-bdd4-b0a29d47ce1a","Type":"ContainerStarted","Data":"7d92941ae8e3465df2cd6c6c994328d5d907511c7ca9a862a616ee1baec55631"}
Sep 17 18:35:36.145726 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:36.145658    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56d5947dc-ft2cl" podStartSLOduration=2.145643159 podStartE2EDuration="2.145643159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:36.14397038 +0000 UTC m=+371.190373306" watchObservedRunningTime="2026-09-17 18:35:36.145643159 +0000 UTC m=+371.192046060"
Sep 17 18:35:45.032447 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:45.032409    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:45.032854 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:45.032469    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:45.037361 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:45.037337    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:45.167898 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:45.167870    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56d5947dc-ft2cl"
Sep 17 18:35:45.231866 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:35:45.231830    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66dd5c4f9b-rr7qm"]
Sep 17 18:36:10.252878 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.252838    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66dd5c4f9b-rr7qm" podUID="c717e2b1-1d25-4231-a367-e2398cbff6fd" containerName="console" containerID="cri-o://02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851" gracePeriod=15
Sep 17 18:36:10.493175 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.493148    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:36:10.610942 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.610833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-oauth-config\") pod \"c717e2b1-1d25-4231-a367-e2398cbff6fd\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") "
Sep 17 18:36:10.610942 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.610884    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-service-ca\") pod \"c717e2b1-1d25-4231-a367-e2398cbff6fd\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") "
Sep 17 18:36:10.610942 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.610932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-serving-cert\") pod \"c717e2b1-1d25-4231-a367-e2398cbff6fd\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") "
Sep 17 18:36:10.611205 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.611048    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-oauth-serving-cert\") pod \"c717e2b1-1d25-4231-a367-e2398cbff6fd\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") "
Sep 17 18:36:10.611205 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.611100    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-trusted-ca-bundle\") pod \"c717e2b1-1d25-4231-a367-e2398cbff6fd\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") "
Sep 17 18:36:10.611205 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.611133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-config\") pod \"c717e2b1-1d25-4231-a367-e2398cbff6fd\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") "
Sep 17 18:36:10.611205 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.611201    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb64d\" (UniqueName: \"kubernetes.io/projected/c717e2b1-1d25-4231-a367-e2398cbff6fd-kube-api-access-rb64d\") pod \"c717e2b1-1d25-4231-a367-e2398cbff6fd\" (UID: \"c717e2b1-1d25-4231-a367-e2398cbff6fd\") "
Sep 17 18:36:10.611408 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.611320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-service-ca" (OuterVolumeSpecName: "service-ca") pod "c717e2b1-1d25-4231-a367-e2398cbff6fd" (UID: "c717e2b1-1d25-4231-a367-e2398cbff6fd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:10.611469 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.611431    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c717e2b1-1d25-4231-a367-e2398cbff6fd" (UID: "c717e2b1-1d25-4231-a367-e2398cbff6fd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:10.611520 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.611495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-config" (OuterVolumeSpecName: "console-config") pod "c717e2b1-1d25-4231-a367-e2398cbff6fd" (UID: "c717e2b1-1d25-4231-a367-e2398cbff6fd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:10.611561 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.611498    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c717e2b1-1d25-4231-a367-e2398cbff6fd" (UID: "c717e2b1-1d25-4231-a367-e2398cbff6fd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:10.611561 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.611507    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-service-ca\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:36:10.611561 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.611548    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-oauth-serving-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:36:10.613151 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.613123    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c717e2b1-1d25-4231-a367-e2398cbff6fd" (UID: "c717e2b1-1d25-4231-a367-e2398cbff6fd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:36:10.613258 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.613156    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c717e2b1-1d25-4231-a367-e2398cbff6fd" (UID: "c717e2b1-1d25-4231-a367-e2398cbff6fd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:36:10.613325 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.613310    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c717e2b1-1d25-4231-a367-e2398cbff6fd-kube-api-access-rb64d" (OuterVolumeSpecName: "kube-api-access-rb64d") pod "c717e2b1-1d25-4231-a367-e2398cbff6fd" (UID: "c717e2b1-1d25-4231-a367-e2398cbff6fd"). InnerVolumeSpecName "kube-api-access-rb64d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:10.712984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.712947    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb64d\" (UniqueName: \"kubernetes.io/projected/c717e2b1-1d25-4231-a367-e2398cbff6fd-kube-api-access-rb64d\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:36:10.712984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.712975    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-oauth-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:36:10.712984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.712984    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-serving-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:36:10.712984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.712995    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-trusted-ca-bundle\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:36:10.713243 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:10.713004    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c717e2b1-1d25-4231-a367-e2398cbff6fd-console-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:36:11.283564 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:11.283531    2577 generic.go:356] "Generic (PLEG): container finished" podID="c717e2b1-1d25-4231-a367-e2398cbff6fd" containerID="02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851" exitCode=0
Sep 17 18:36:11.283953 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:11.283576    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66dd5c4f9b-rr7qm"
Sep 17 18:36:11.283953 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:11.283576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66dd5c4f9b-rr7qm" event={"ID":"c717e2b1-1d25-4231-a367-e2398cbff6fd","Type":"ContainerDied","Data":"02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851"}
Sep 17 18:36:11.283953 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:11.283695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66dd5c4f9b-rr7qm" event={"ID":"c717e2b1-1d25-4231-a367-e2398cbff6fd","Type":"ContainerDied","Data":"3c236007e42f4e3e0f32802590c30f1b58eb4dbd2e2dd1761bde6ffc5aa5f3fb"}
Sep 17 18:36:11.283953 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:11.283721    2577 scope.go:117] "RemoveContainer" containerID="02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851"
Sep 17 18:36:11.292126 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:11.292108    2577 scope.go:117] "RemoveContainer" containerID="02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851"
Sep 17 18:36:11.292404 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:36:11.292385    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851\": container with ID starting with 02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851 not found: ID does not exist" containerID="02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851"
Sep 17 18:36:11.292450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:11.292415    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851"} err="failed to get container status \"02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851\": rpc error: code = NotFound desc = could not find container \"02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851\": container with ID starting with 02d5aa5e5d55b7364ad2f6298a4097044d6768046176a6ad0631f457c8654851 not found: ID does not exist"
Sep 17 18:36:11.300772 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:11.300749    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66dd5c4f9b-rr7qm"]
Sep 17 18:36:11.304973 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:11.304952    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66dd5c4f9b-rr7qm"]
Sep 17 18:36:13.238193 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:13.238160    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c717e2b1-1d25-4231-a367-e2398cbff6fd" path="/var/lib/kubelet/pods/c717e2b1-1d25-4231-a367-e2398cbff6fd/volumes"
Sep 17 18:36:15.811367 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.811331    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-48tw5"]
Sep 17 18:36:15.811797 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.811672    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c717e2b1-1d25-4231-a367-e2398cbff6fd" containerName="console"
Sep 17 18:36:15.811797 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.811683    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c717e2b1-1d25-4231-a367-e2398cbff6fd" containerName="console"
Sep 17 18:36:15.811797 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.811760    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c717e2b1-1d25-4231-a367-e2398cbff6fd" containerName="console"
Sep 17 18:36:15.813961 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.813942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:15.817646 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.817606    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-67qnp\""
Sep 17 18:36:15.817798 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.817778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 18:36:15.827664 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.827638    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-48tw5"]
Sep 17 18:36:15.833994 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.833970    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-fkr56"]
Sep 17 18:36:15.836993 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.836973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:15.841535 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.841518    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-qztgb\""
Sep 17 18:36:15.841640 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.841519    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 18:36:15.841640 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.841519    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 18:36:15.852256 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.852233    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-fkr56"]
Sep 17 18:36:15.959192 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.959153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j5nl\" (UniqueName: \"kubernetes.io/projected/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-kube-api-access-2j5nl\") pod \"model-serving-api-7b954ffc95-48tw5\" (UID: \"fabfa745-3eaf-4d62-b9f2-895cc8e9fa35\") " pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:15.959192 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.959194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2v5\" (UniqueName: \"kubernetes.io/projected/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-kube-api-access-lf2v5\") pod \"odh-model-controller-7b966d7984-fkr56\" (UID: \"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013\") " pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:15.959432 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.959231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-metrics-certs\") pod \"odh-model-controller-7b966d7984-fkr56\" (UID: \"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013\") " pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:15.959432 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.959254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-cert\") pod \"odh-model-controller-7b966d7984-fkr56\" (UID: \"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013\") " pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:15.959432 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:15.959278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-tls-certs\") pod \"model-serving-api-7b954ffc95-48tw5\" (UID: \"fabfa745-3eaf-4d62-b9f2-895cc8e9fa35\") " pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:16.060641 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.060602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j5nl\" (UniqueName: \"kubernetes.io/projected/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-kube-api-access-2j5nl\") pod \"model-serving-api-7b954ffc95-48tw5\" (UID: \"fabfa745-3eaf-4d62-b9f2-895cc8e9fa35\") " pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:16.060641 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.060639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2v5\" (UniqueName: \"kubernetes.io/projected/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-kube-api-access-lf2v5\") pod \"odh-model-controller-7b966d7984-fkr56\" (UID: \"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013\") " pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:16.060934 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.060778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-metrics-certs\") pod \"odh-model-controller-7b966d7984-fkr56\" (UID: \"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013\") " pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:16.060934 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.060834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-cert\") pod \"odh-model-controller-7b966d7984-fkr56\" (UID: \"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013\") " pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:16.060934 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.060865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-tls-certs\") pod \"model-serving-api-7b954ffc95-48tw5\" (UID: \"fabfa745-3eaf-4d62-b9f2-895cc8e9fa35\") " pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:16.061096 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:36:16.060975    2577 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 18:36:16.061096 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:36:16.061014    2577 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 18:36:16.061096 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:36:16.061042    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-cert podName:a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013 nodeName:}" failed. No retries permitted until 2026-09-17 18:36:16.561021621 +0000 UTC m=+411.607424505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-cert") pod "odh-model-controller-7b966d7984-fkr56" (UID: "a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013") : secret "odh-model-controller-webhook-cert" not found
Sep 17 18:36:16.061096 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:36:16.061082    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-tls-certs podName:fabfa745-3eaf-4d62-b9f2-895cc8e9fa35 nodeName:}" failed. No retries permitted until 2026-09-17 18:36:16.56106406 +0000 UTC m=+411.607466958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-tls-certs") pod "model-serving-api-7b954ffc95-48tw5" (UID: "fabfa745-3eaf-4d62-b9f2-895cc8e9fa35") : secret "model-serving-api-tls" not found
Sep 17 18:36:16.063336 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.063277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-metrics-certs\") pod \"odh-model-controller-7b966d7984-fkr56\" (UID: \"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013\") " pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:16.069532 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.069506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j5nl\" (UniqueName: \"kubernetes.io/projected/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-kube-api-access-2j5nl\") pod \"model-serving-api-7b954ffc95-48tw5\" (UID: \"fabfa745-3eaf-4d62-b9f2-895cc8e9fa35\") " pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:16.069670 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.069511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2v5\" (UniqueName: \"kubernetes.io/projected/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-kube-api-access-lf2v5\") pod \"odh-model-controller-7b966d7984-fkr56\" (UID: \"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013\") " pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:16.565165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.565127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-cert\") pod \"odh-model-controller-7b966d7984-fkr56\" (UID: \"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013\") " pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:16.565165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.565170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-tls-certs\") pod \"model-serving-api-7b954ffc95-48tw5\" (UID: \"fabfa745-3eaf-4d62-b9f2-895cc8e9fa35\") " pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:16.565384 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:36:16.565271    2577 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 18:36:16.565384 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:36:16.565336    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-tls-certs podName:fabfa745-3eaf-4d62-b9f2-895cc8e9fa35 nodeName:}" failed. No retries permitted until 2026-09-17 18:36:17.56531949 +0000 UTC m=+412.611722387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-tls-certs") pod "model-serving-api-7b954ffc95-48tw5" (UID: "fabfa745-3eaf-4d62-b9f2-895cc8e9fa35") : secret "model-serving-api-tls" not found
Sep 17 18:36:16.567553 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.567528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013-cert\") pod \"odh-model-controller-7b966d7984-fkr56\" (UID: \"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013\") " pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:16.747379 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:16.747337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:17.019638 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:17.019600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-fkr56"]
Sep 17 18:36:17.022643 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:36:17.022620    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69ebe86_7eb1_40aa_bf0f_2fa89a8a2013.slice/crio-146057b6840ca13c8480fa2c1ea5ff63adc8a79ddf8325befae0025f4dc50d55 WatchSource:0}: Error finding container 146057b6840ca13c8480fa2c1ea5ff63adc8a79ddf8325befae0025f4dc50d55: Status 404 returned error can't find the container with id 146057b6840ca13c8480fa2c1ea5ff63adc8a79ddf8325befae0025f4dc50d55
Sep 17 18:36:17.311986 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:17.311887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-fkr56" event={"ID":"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013","Type":"ContainerStarted","Data":"146057b6840ca13c8480fa2c1ea5ff63adc8a79ddf8325befae0025f4dc50d55"}
Sep 17 18:36:17.573670 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:17.573577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-tls-certs\") pod \"model-serving-api-7b954ffc95-48tw5\" (UID: \"fabfa745-3eaf-4d62-b9f2-895cc8e9fa35\") " pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:17.576284 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:17.576259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabfa745-3eaf-4d62-b9f2-895cc8e9fa35-tls-certs\") pod \"model-serving-api-7b954ffc95-48tw5\" (UID: \"fabfa745-3eaf-4d62-b9f2-895cc8e9fa35\") " pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:17.625257 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:17.625207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:17.902076 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:17.902036    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-48tw5"]
Sep 17 18:36:17.903972 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:36:17.903943    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabfa745_3eaf_4d62_b9f2_895cc8e9fa35.slice/crio-192f093c18361066265e9687c822e4b35aa6aa241c95658b98dad74c3979bbd3 WatchSource:0}: Error finding container 192f093c18361066265e9687c822e4b35aa6aa241c95658b98dad74c3979bbd3: Status 404 returned error can't find the container with id 192f093c18361066265e9687c822e4b35aa6aa241c95658b98dad74c3979bbd3
Sep 17 18:36:18.324960 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:18.324886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-48tw5" event={"ID":"fabfa745-3eaf-4d62-b9f2-895cc8e9fa35","Type":"ContainerStarted","Data":"192f093c18361066265e9687c822e4b35aa6aa241c95658b98dad74c3979bbd3"}
Sep 17 18:36:21.345154 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:21.345116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-fkr56" event={"ID":"a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013","Type":"ContainerStarted","Data":"c7df321eef9c8662ef2a1b3c0a71edfe172554260e2b058d1a153d10a44d93b5"}
Sep 17 18:36:21.345604 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:21.345222    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:21.346555 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:21.346534    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-48tw5" event={"ID":"fabfa745-3eaf-4d62-b9f2-895cc8e9fa35","Type":"ContainerStarted","Data":"a1120939c4a7a669b69c7b2b3def4ca8453021a4e8afe99cca2da9c94c22a55f"}
Sep 17 18:36:21.346680 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:21.346665    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:36:21.366584 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:21.366538    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-fkr56" podStartSLOduration=2.508651193 podStartE2EDuration="6.366524825s" totalImagesPullingTime="3.857873632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:21.365484231 +0000 UTC m=+416.411887172" watchObservedRunningTime="2026-09-17 18:36:21.366524825 +0000 UTC m=+416.412927726"
Sep 17 18:36:21.384368 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:21.384316    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-48tw5" podStartSLOduration=3.399925508 podStartE2EDuration="6.384299602s" totalImagesPullingTime="2.984374094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:21.382476724 +0000 UTC m=+416.428879665" watchObservedRunningTime="2026-09-17 18:36:21.384299602 +0000 UTC m=+416.430702503"
Sep 17 18:36:32.354626 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:32.354595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-fkr56"
Sep 17 18:36:32.356361 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:36:32.356340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-48tw5"
Sep 17 18:37:52.937699 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:52.937666    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"]
Sep 17 18:37:52.940060 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:52.940043    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"
Sep 17 18:37:52.942633 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:52.942605    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 18:37:52.942771 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:52.942684    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-j4mg9\""
Sep 17 18:37:52.949355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:52.949327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"]
Sep 17 18:37:52.980457 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:52.980428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mhbwp\" (UID: \"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"
Sep 17 18:37:52.980632 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:52.980496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvw4c\" (UniqueName: \"kubernetes.io/projected/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-kube-api-access-vvw4c\") pod \"seaweedfs-tls-custom-6644bf5d6c-mhbwp\" (UID: \"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"
Sep 17 18:37:53.081664 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:53.081620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvw4c\" (UniqueName: \"kubernetes.io/projected/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-kube-api-access-vvw4c\") pod \"seaweedfs-tls-custom-6644bf5d6c-mhbwp\" (UID: \"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"
Sep 17 18:37:53.081852 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:53.081709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mhbwp\" (UID: \"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"
Sep 17 18:37:53.082181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:53.082162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mhbwp\" (UID: \"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"
Sep 17 18:37:53.090289 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:53.090253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvw4c\" (UniqueName: \"kubernetes.io/projected/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-kube-api-access-vvw4c\") pod \"seaweedfs-tls-custom-6644bf5d6c-mhbwp\" (UID: \"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"
Sep 17 18:37:53.250894 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:53.250803    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"
Sep 17 18:37:53.518538 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:53.518507    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"]
Sep 17 18:37:53.521057 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:37:53.521031    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e7d079_d9fa_41bf_94c5_d5eab2c7ab80.slice/crio-c187c2fa07acbc695e622d78a30b5caa4a97b1fe4d9043b4148487812ab9e910 WatchSource:0}: Error finding container c187c2fa07acbc695e622d78a30b5caa4a97b1fe4d9043b4148487812ab9e910: Status 404 returned error can't find the container with id c187c2fa07acbc695e622d78a30b5caa4a97b1fe4d9043b4148487812ab9e910
Sep 17 18:37:53.779798 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:53.779708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp" event={"ID":"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80","Type":"ContainerStarted","Data":"c187c2fa07acbc695e622d78a30b5caa4a97b1fe4d9043b4148487812ab9e910"}
Sep 17 18:37:56.797224 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:56.797183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp" event={"ID":"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80","Type":"ContainerStarted","Data":"bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007"}
Sep 17 18:37:56.814309 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:56.814247    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp" podStartSLOduration=2.203014345 podStartE2EDuration="4.814231604s" totalImagesPullingTime="2.611217259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:56.812310953 +0000 UTC m=+511.858713854" watchObservedRunningTime="2026-09-17 18:37:56.814231604 +0000 UTC m=+511.860634505"
Sep 17 18:37:58.536693 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:58.536653    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"]
Sep 17 18:37:58.806096 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:37:58.805997    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp" podUID="48e7d079-d9fa-41bf-94c5-d5eab2c7ab80" containerName="seaweedfs-tls-custom" containerID="cri-o://bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007" gracePeriod=30
Sep 17 18:38:00.050727 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.050700    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"
Sep 17 18:38:00.151524 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.151432    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvw4c\" (UniqueName: \"kubernetes.io/projected/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-kube-api-access-vvw4c\") pod \"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80\" (UID: \"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80\") "
Sep 17 18:38:00.151524 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.151479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-data\") pod \"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80\" (UID: \"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80\") "
Sep 17 18:38:00.152734 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.152710    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-data" (OuterVolumeSpecName: "data") pod "48e7d079-d9fa-41bf-94c5-d5eab2c7ab80" (UID: "48e7d079-d9fa-41bf-94c5-d5eab2c7ab80"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:00.153551 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.153529    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-kube-api-access-vvw4c" (OuterVolumeSpecName: "kube-api-access-vvw4c") pod "48e7d079-d9fa-41bf-94c5-d5eab2c7ab80" (UID: "48e7d079-d9fa-41bf-94c5-d5eab2c7ab80"). InnerVolumeSpecName "kube-api-access-vvw4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:00.252273 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.252231    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvw4c\" (UniqueName: \"kubernetes.io/projected/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-kube-api-access-vvw4c\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:38:00.252273 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.252267    2577 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80-data\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:38:00.816102 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.816068    2577 generic.go:356] "Generic (PLEG): container finished" podID="48e7d079-d9fa-41bf-94c5-d5eab2c7ab80" containerID="bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007" exitCode=0
Sep 17 18:38:00.816102 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.816108    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"
Sep 17 18:38:00.816321 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.816118    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp" event={"ID":"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80","Type":"ContainerDied","Data":"bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007"}
Sep 17 18:38:00.816321 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.816159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp" event={"ID":"48e7d079-d9fa-41bf-94c5-d5eab2c7ab80","Type":"ContainerDied","Data":"c187c2fa07acbc695e622d78a30b5caa4a97b1fe4d9043b4148487812ab9e910"}
Sep 17 18:38:00.816321 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.816179    2577 scope.go:117] "RemoveContainer" containerID="bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007"
Sep 17 18:38:00.829932 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.829630    2577 scope.go:117] "RemoveContainer" containerID="bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007"
Sep 17 18:38:00.830253 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:38:00.830228    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007\": container with ID starting with bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007 not found: ID does not exist" containerID="bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007"
Sep 17 18:38:00.830324 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.830270    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007"} err="failed to get container status \"bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007\": rpc error: code = NotFound desc = could not find container \"bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007\": container with ID starting with bdcb6a9c731b3e4640887ccd8881918163fc004223bee9271291567f181bd007 not found: ID does not exist"
Sep 17 18:38:00.843517 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.843492    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"]
Sep 17 18:38:00.853262 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:00.853237    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mhbwp"]
Sep 17 18:38:01.238557 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:01.238523    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48e7d079-d9fa-41bf-94c5-d5eab2c7ab80" path="/var/lib/kubelet/pods/48e7d079-d9fa-41bf-94c5-d5eab2c7ab80/volumes"
Sep 17 18:38:41.349338 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.349302    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"]
Sep 17 18:38:41.349824 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.349716    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e7d079-d9fa-41bf-94c5-d5eab2c7ab80" containerName="seaweedfs-tls-custom"
Sep 17 18:38:41.349824 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.349731    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e7d079-d9fa-41bf-94c5-d5eab2c7ab80" containerName="seaweedfs-tls-custom"
Sep 17 18:38:41.349824 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.349808    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e7d079-d9fa-41bf-94c5-d5eab2c7ab80" containerName="seaweedfs-tls-custom"
Sep 17 18:38:41.352824 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.352809    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.355845 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.355820    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:38:41.356352 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.356316    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4585ab-predictor-serving-cert\""
Sep 17 18:38:41.356352 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.356319    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-4585ab\""
Sep 17 18:38:41.356513 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.356431    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:38:41.356579 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.356560    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:38:41.356632 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.356565    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4585ab-kube-rbac-proxy-sar-config\""
Sep 17 18:38:41.356632 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.356621    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-4585ab-dockercfg-8kf4b\""
Sep 17 18:38:41.367387 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.367364    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"]
Sep 17 18:38:41.399841 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.399803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm254\" (UniqueName: \"kubernetes.io/projected/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-api-access-dm254\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.400006 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.399860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a754161-f6a6-41c3-97fd-fe9f3e618516-kserve-provision-location\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.400006 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.399944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-cabundle-cert\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.400091 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.400010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a754161-f6a6-41c3-97fd-fe9f3e618516-proxy-tls\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.400091 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.400042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.500399 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.500362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-cabundle-cert\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.500598 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.500442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a754161-f6a6-41c3-97fd-fe9f3e618516-proxy-tls\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.500598 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.500471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.500598 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.500498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm254\" (UniqueName: \"kubernetes.io/projected/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-api-access-dm254\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.500598 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.500535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a754161-f6a6-41c3-97fd-fe9f3e618516-kserve-provision-location\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.500826 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:38:41.500610    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-serving-cert: secret "isvc-init-fail-4585ab-predictor-serving-cert" not found
Sep 17 18:38:41.500826 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:38:41.500741    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a754161-f6a6-41c3-97fd-fe9f3e618516-proxy-tls podName:4a754161-f6a6-41c3-97fd-fe9f3e618516 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:42.000719626 +0000 UTC m=+557.047122509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a754161-f6a6-41c3-97fd-fe9f3e618516-proxy-tls") pod "isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs" (UID: "4a754161-f6a6-41c3-97fd-fe9f3e618516") : secret "isvc-init-fail-4585ab-predictor-serving-cert" not found
Sep 17 18:38:41.500964 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.500926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a754161-f6a6-41c3-97fd-fe9f3e618516-kserve-provision-location\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.501168 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.501148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-cabundle-cert\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.501223 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.501165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:41.509167 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:41.509141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm254\" (UniqueName: \"kubernetes.io/projected/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-api-access-dm254\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:42.005383 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:42.005351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a754161-f6a6-41c3-97fd-fe9f3e618516-proxy-tls\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:42.007788 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:42.007756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a754161-f6a6-41c3-97fd-fe9f3e618516-proxy-tls\") pod \"isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") " pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:42.263162 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:42.263073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:42.539749 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:42.539727    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"]
Sep 17 18:38:42.542060 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:38:42.542031    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a754161_f6a6_41c3_97fd_fe9f3e618516.slice/crio-4f23a36e9fa99869bde8cfc4a2f1c06800f37406a270376859dd8def3b26e760 WatchSource:0}: Error finding container 4f23a36e9fa99869bde8cfc4a2f1c06800f37406a270376859dd8def3b26e760: Status 404 returned error can't find the container with id 4f23a36e9fa99869bde8cfc4a2f1c06800f37406a270376859dd8def3b26e760
Sep 17 18:38:43.009645 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:43.009605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs" event={"ID":"4a754161-f6a6-41c3-97fd-fe9f3e618516","Type":"ContainerStarted","Data":"4f23a36e9fa99869bde8cfc4a2f1c06800f37406a270376859dd8def3b26e760"}
Sep 17 18:38:47.033654 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:47.033617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs" event={"ID":"4a754161-f6a6-41c3-97fd-fe9f3e618516","Type":"ContainerStarted","Data":"fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8"}
Sep 17 18:38:52.057234 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:52.057207    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs_4a754161-f6a6-41c3-97fd-fe9f3e618516/storage-initializer/0.log"
Sep 17 18:38:52.057577 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:52.057245    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a754161-f6a6-41c3-97fd-fe9f3e618516" containerID="fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8" exitCode=1
Sep 17 18:38:52.057577 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:52.057326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs" event={"ID":"4a754161-f6a6-41c3-97fd-fe9f3e618516","Type":"ContainerDied","Data":"fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8"}
Sep 17 18:38:53.064144 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:53.064112    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs_4a754161-f6a6-41c3-97fd-fe9f3e618516/storage-initializer/0.log"
Sep 17 18:38:53.064541 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:53.064164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs" event={"ID":"4a754161-f6a6-41c3-97fd-fe9f3e618516","Type":"ContainerStarted","Data":"1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d"}
Sep 17 18:38:56.015618 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:56.015586    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"]
Sep 17 18:38:56.016090 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:56.015818    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs" podUID="4a754161-f6a6-41c3-97fd-fe9f3e618516" containerName="storage-initializer" containerID="cri-o://1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d" gracePeriod=30
Sep 17 18:38:57.364365 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.364342    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs_4a754161-f6a6-41c3-97fd-fe9f3e618516/storage-initializer/1.log"
Sep 17 18:38:57.364756 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.364741    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs_4a754161-f6a6-41c3-97fd-fe9f3e618516/storage-initializer/0.log"
Sep 17 18:38:57.367065 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.367049    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:57.453877 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.453847    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a754161-f6a6-41c3-97fd-fe9f3e618516-kserve-provision-location\") pod \"4a754161-f6a6-41c3-97fd-fe9f3e618516\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") "
Sep 17 18:38:57.454073 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.453887    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-cabundle-cert\") pod \"4a754161-f6a6-41c3-97fd-fe9f3e618516\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") "
Sep 17 18:38:57.454073 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.453925    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-rbac-proxy-sar-config\") pod \"4a754161-f6a6-41c3-97fd-fe9f3e618516\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") "
Sep 17 18:38:57.454073 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.453952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a754161-f6a6-41c3-97fd-fe9f3e618516-proxy-tls\") pod \"4a754161-f6a6-41c3-97fd-fe9f3e618516\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") "
Sep 17 18:38:57.454073 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.453999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm254\" (UniqueName: \"kubernetes.io/projected/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-api-access-dm254\") pod \"4a754161-f6a6-41c3-97fd-fe9f3e618516\" (UID: \"4a754161-f6a6-41c3-97fd-fe9f3e618516\") "
Sep 17 18:38:57.454301 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.454193    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a754161-f6a6-41c3-97fd-fe9f3e618516-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a754161-f6a6-41c3-97fd-fe9f3e618516" (UID: "4a754161-f6a6-41c3-97fd-fe9f3e618516"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:57.454301 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.454278    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a754161-f6a6-41c3-97fd-fe9f3e618516" (UID: "4a754161-f6a6-41c3-97fd-fe9f3e618516"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:38:57.454301 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.454290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a754161-f6a6-41c3-97fd-fe9f3e618516" (UID: "4a754161-f6a6-41c3-97fd-fe9f3e618516"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:38:57.456153 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.456132    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a754161-f6a6-41c3-97fd-fe9f3e618516-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a754161-f6a6-41c3-97fd-fe9f3e618516" (UID: "4a754161-f6a6-41c3-97fd-fe9f3e618516"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:57.456257 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.456237    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-api-access-dm254" (OuterVolumeSpecName: "kube-api-access-dm254") pod "4a754161-f6a6-41c3-97fd-fe9f3e618516" (UID: "4a754161-f6a6-41c3-97fd-fe9f3e618516"). InnerVolumeSpecName "kube-api-access-dm254". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:57.555101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.555065    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a754161-f6a6-41c3-97fd-fe9f3e618516-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:38:57.555101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.555098    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm254\" (UniqueName: \"kubernetes.io/projected/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-api-access-dm254\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:38:57.555101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.555109    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a754161-f6a6-41c3-97fd-fe9f3e618516-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:38:57.555348 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.555117    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:38:57.555348 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:57.555126    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a754161-f6a6-41c3-97fd-fe9f3e618516-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:38:58.090511 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.090483    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs_4a754161-f6a6-41c3-97fd-fe9f3e618516/storage-initializer/1.log"
Sep 17 18:38:58.090897 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.090878    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs_4a754161-f6a6-41c3-97fd-fe9f3e618516/storage-initializer/0.log"
Sep 17 18:38:58.091006 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.090933    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a754161-f6a6-41c3-97fd-fe9f3e618516" containerID="1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d" exitCode=1
Sep 17 18:38:58.091047 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.091005    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"
Sep 17 18:38:58.091047 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.091015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs" event={"ID":"4a754161-f6a6-41c3-97fd-fe9f3e618516","Type":"ContainerDied","Data":"1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d"}
Sep 17 18:38:58.091124 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.091070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs" event={"ID":"4a754161-f6a6-41c3-97fd-fe9f3e618516","Type":"ContainerDied","Data":"4f23a36e9fa99869bde8cfc4a2f1c06800f37406a270376859dd8def3b26e760"}
Sep 17 18:38:58.091124 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.091097    2577 scope.go:117] "RemoveContainer" containerID="1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d"
Sep 17 18:38:58.100843 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.100823    2577 scope.go:117] "RemoveContainer" containerID="fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8"
Sep 17 18:38:58.108513 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.108497    2577 scope.go:117] "RemoveContainer" containerID="1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d"
Sep 17 18:38:58.108818 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:38:58.108800    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d\": container with ID starting with 1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d not found: ID does not exist" containerID="1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d"
Sep 17 18:38:58.108867 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.108826    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d"} err="failed to get container status \"1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d\": rpc error: code = NotFound desc = could not find container \"1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d\": container with ID starting with 1930010d3684ba5fcfd7489bf4af6017131d88e8499d6a6cb453fa9918a9c07d not found: ID does not exist"
Sep 17 18:38:58.108867 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.108843    2577 scope.go:117] "RemoveContainer" containerID="fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8"
Sep 17 18:38:58.109080 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:38:58.109064    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8\": container with ID starting with fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8 not found: ID does not exist" containerID="fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8"
Sep 17 18:38:58.109118 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.109083    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8"} err="failed to get container status \"fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8\": rpc error: code = NotFound desc = could not find container \"fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8\": container with ID starting with fa561b32bd046b8c107d866e0a058d9c2fc07bc76f4edee7ba4f2f6a745fc2c8 not found: ID does not exist"
Sep 17 18:38:58.129765 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.129729    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"]
Sep 17 18:38:58.133168 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:58.133145    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4585ab-predictor-9bcf589f9-fmgqs"]
Sep 17 18:38:59.238109 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:38:59.238075    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a754161-f6a6-41c3-97fd-fe9f3e618516" path="/var/lib/kubelet/pods/4a754161-f6a6-41c3-97fd-fe9f3e618516/volumes"
Sep 17 18:39:01.238895 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.238859    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"]
Sep 17 18:39:01.239268 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.239257    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a754161-f6a6-41c3-97fd-fe9f3e618516" containerName="storage-initializer"
Sep 17 18:39:01.239309 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.239271    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a754161-f6a6-41c3-97fd-fe9f3e618516" containerName="storage-initializer"
Sep 17 18:39:01.239309 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.239294    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a754161-f6a6-41c3-97fd-fe9f3e618516" containerName="storage-initializer"
Sep 17 18:39:01.239309 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.239300    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a754161-f6a6-41c3-97fd-fe9f3e618516" containerName="storage-initializer"
Sep 17 18:39:01.239399 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.239359    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a754161-f6a6-41c3-97fd-fe9f3e618516" containerName="storage-initializer"
Sep 17 18:39:01.239485 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.239473    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a754161-f6a6-41c3-97fd-fe9f3e618516" containerName="storage-initializer"
Sep 17 18:39:01.244436 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.244419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.247201 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.247179    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:39:01.247201 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.247191    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 18:39:01.247393 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.247230    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:39:01.247393 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.247309    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 18:39:01.248112 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.248093    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:39:01.248449 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.248123    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:39:01.248449 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.248131    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 18:39:01.250809 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.250775    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"]
Sep 17 18:39:01.389307 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.389266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.389475 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.389324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.389475 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.389391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.389545 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.389475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.389545 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.389511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjt9g\" (UniqueName: \"kubernetes.io/projected/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-api-access-fjt9g\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.490735 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.490637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjt9g\" (UniqueName: \"kubernetes.io/projected/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-api-access-fjt9g\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.490735 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.490719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.491029 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.490743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.491029 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.490759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.491029 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.490790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.491029 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:39:01.490871    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Sep 17 18:39:01.491029 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:39:01.491005    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-proxy-tls podName:69a047e7-ff7a-4535-9caf-aabe5e2ae9fb nodeName:}" failed. No retries permitted until 2026-09-17 18:39:01.99097142 +0000 UTC m=+577.037374299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" (UID: "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Sep 17 18:39:01.491313 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.491290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.491678 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.491656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.491723 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.491656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.507130 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.507092    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjt9g\" (UniqueName: \"kubernetes.io/projected/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-api-access-fjt9g\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.996772 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.996738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:01.999251 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:01.999226    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zs5j6\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:02.155723 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:02.155671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:02.435856 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:02.435832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"]
Sep 17 18:39:02.437515 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:39:02.437480    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a047e7_ff7a_4535_9caf_aabe5e2ae9fb.slice/crio-9d9e62c21412349352cd288d944fb74bc82862c853c8b8b8dad5cba73190322d WatchSource:0}: Error finding container 9d9e62c21412349352cd288d944fb74bc82862c853c8b8b8dad5cba73190322d: Status 404 returned error can't find the container with id 9d9e62c21412349352cd288d944fb74bc82862c853c8b8b8dad5cba73190322d
Sep 17 18:39:03.124624 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:03.124587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" event={"ID":"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb","Type":"ContainerStarted","Data":"721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49"}
Sep 17 18:39:03.124819 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:03.124652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" event={"ID":"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb","Type":"ContainerStarted","Data":"9d9e62c21412349352cd288d944fb74bc82862c853c8b8b8dad5cba73190322d"}
Sep 17 18:39:07.143687 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:07.143652    2577 generic.go:356] "Generic (PLEG): container finished" podID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerID="721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49" exitCode=0
Sep 17 18:39:07.144092 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:07.143727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" event={"ID":"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb","Type":"ContainerDied","Data":"721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49"}
Sep 17 18:39:31.287776 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:31.287733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" event={"ID":"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb","Type":"ContainerStarted","Data":"a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f"}
Sep 17 18:39:34.305154 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:34.305119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" event={"ID":"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb","Type":"ContainerStarted","Data":"4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c"}
Sep 17 18:39:34.305543 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:34.305265    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:34.341019 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:34.340973    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podStartSLOduration=3.477907927 podStartE2EDuration="33.340958371s" totalImagesPullingTime="26.044333086s" totalInitContainerRuntime="3.818717358s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:34.339131068 +0000 UTC m=+609.385534124" watchObservedRunningTime="2026-09-17 18:39:34.340958371 +0000 UTC m=+609.387361273"
Sep 17 18:39:35.310283 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:35.310246    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:35.311635 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:35.311599    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:39:36.314924 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:36.314871    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:39:41.319509 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:41.319476    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:39:41.320040 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:41.319993    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:39:51.320203 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:39:51.320161    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:40:01.320436 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:40:01.320390    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:40:11.320059 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:40:11.320018    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:40:21.320469 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:40:21.320428    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:40:31.320449 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:40:31.320399    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:40:41.320784 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:40:41.320744    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:40:45.233708 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:40:45.233655    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:40:55.238772 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:40:55.238739    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:41:01.801583 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:01.801549    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"]
Sep 17 18:41:01.802016 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:01.801778    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" containerID="cri-o://a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f" gracePeriod=30
Sep 17 18:41:01.802016 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:01.801797    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kube-rbac-proxy" containerID="cri-o://4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c" gracePeriod=30
Sep 17 18:41:02.731155 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:02.731116    2577 generic.go:356] "Generic (PLEG): container finished" podID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerID="4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c" exitCode=2
Sep 17 18:41:02.731398 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:02.731185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" event={"ID":"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb","Type":"ContainerDied","Data":"4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c"}
Sep 17 18:41:05.234242 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:05.234186    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:41:06.315260 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:06.315219    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 18:41:06.987603 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:06.987573    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"]
Sep 17 18:41:06.990066 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:06.990043    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:06.992947 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:06.992880    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 18:41:06.993204 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:06.993180    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 18:41:07.009930 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.009880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"]
Sep 17 18:41:07.039487 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.039439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.039487 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.039483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afce11c7-5af4-47d4-9040-f6ac4273712d-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.039738 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.039547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afce11c7-5af4-47d4-9040-f6ac4273712d-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.039738 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.039582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.039738 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.039612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjvc\" (UniqueName: \"kubernetes.io/projected/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-api-access-lrjvc\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.140181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.140153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.140318 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.140212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjvc\" (UniqueName: \"kubernetes.io/projected/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-api-access-lrjvc\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.140318 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.140295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.140426 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.140324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afce11c7-5af4-47d4-9040-f6ac4273712d-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.140426 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.140379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afce11c7-5af4-47d4-9040-f6ac4273712d-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.140732 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.140709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afce11c7-5af4-47d4-9040-f6ac4273712d-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.140837 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.140818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.141016 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.140987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.142816 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.142795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afce11c7-5af4-47d4-9040-f6ac4273712d-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.148448 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.148423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjvc\" (UniqueName: \"kubernetes.io/projected/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-api-access-lrjvc\") pod \"isvc-predictive-xgboost-predictor-695f978978-zrfg9\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.163355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.163325    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:41:07.241333 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.241254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kserve-provision-location\") pod \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") "
Sep 17 18:41:07.241333 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.241300    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-rbac-proxy-sar-config\") pod \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") "
Sep 17 18:41:07.241333 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.241330    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjt9g\" (UniqueName: \"kubernetes.io/projected/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-api-access-fjt9g\") pod \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") "
Sep 17 18:41:07.241613 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.241347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-cabundle-cert\") pod \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") "
Sep 17 18:41:07.241613 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.241405    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-proxy-tls\") pod \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\" (UID: \"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb\") "
Sep 17 18:41:07.241711 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.241621    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" (UID: "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:07.241775 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.241751    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" (UID: "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:07.241822 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.241755    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" (UID: "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:07.243491 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.243468    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" (UID: "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:07.243597 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.243577    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-api-access-fjt9g" (OuterVolumeSpecName: "kube-api-access-fjt9g") pod "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" (UID: "69a047e7-ff7a-4535-9caf-aabe5e2ae9fb"). InnerVolumeSpecName "kube-api-access-fjt9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:07.342717 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.342677    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:41:07.342717 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.342710    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:41:07.342717 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.342719    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:41:07.342717 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.342728    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjt9g\" (UniqueName: \"kubernetes.io/projected/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-kube-api-access-fjt9g\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:41:07.343203 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.342737    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:41:07.367033 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.366993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:07.660569 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.660542    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"]
Sep 17 18:41:07.661987 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:41:07.661957    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafce11c7_5af4_47d4_9040_f6ac4273712d.slice/crio-5b43ab71332a93eaac4530df40767231fd571323d06256209e08446fadb8455d WatchSource:0}: Error finding container 5b43ab71332a93eaac4530df40767231fd571323d06256209e08446fadb8455d: Status 404 returned error can't find the container with id 5b43ab71332a93eaac4530df40767231fd571323d06256209e08446fadb8455d
Sep 17 18:41:07.758734 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.758688    2577 generic.go:356] "Generic (PLEG): container finished" podID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerID="a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f" exitCode=0
Sep 17 18:41:07.758943 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.758755    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"
Sep 17 18:41:07.758943 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.758768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" event={"ID":"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb","Type":"ContainerDied","Data":"a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f"}
Sep 17 18:41:07.758943 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.758865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6" event={"ID":"69a047e7-ff7a-4535-9caf-aabe5e2ae9fb","Type":"ContainerDied","Data":"9d9e62c21412349352cd288d944fb74bc82862c853c8b8b8dad5cba73190322d"}
Sep 17 18:41:07.758943 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.758928    2577 scope.go:117] "RemoveContainer" containerID="4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c"
Sep 17 18:41:07.760841 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.760815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" event={"ID":"afce11c7-5af4-47d4-9040-f6ac4273712d","Type":"ContainerStarted","Data":"2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801"}
Sep 17 18:41:07.760993 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.760877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" event={"ID":"afce11c7-5af4-47d4-9040-f6ac4273712d","Type":"ContainerStarted","Data":"5b43ab71332a93eaac4530df40767231fd571323d06256209e08446fadb8455d"}
Sep 17 18:41:07.769077 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.769054    2577 scope.go:117] "RemoveContainer" containerID="a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f"
Sep 17 18:41:07.777081 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.777062    2577 scope.go:117] "RemoveContainer" containerID="721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49"
Sep 17 18:41:07.785960 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.785938    2577 scope.go:117] "RemoveContainer" containerID="4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c"
Sep 17 18:41:07.786309 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:41:07.786283    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c\": container with ID starting with 4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c not found: ID does not exist" containerID="4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c"
Sep 17 18:41:07.786400 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.786315    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c"} err="failed to get container status \"4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c\": rpc error: code = NotFound desc = could not find container \"4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c\": container with ID starting with 4ad6232f524137d52326eb6ce25584d7b17063b43a7e3be527c234330db5977c not found: ID does not exist"
Sep 17 18:41:07.786400 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.786332    2577 scope.go:117] "RemoveContainer" containerID="a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f"
Sep 17 18:41:07.786600 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:41:07.786582    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f\": container with ID starting with a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f not found: ID does not exist" containerID="a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f"
Sep 17 18:41:07.786667 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.786609    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f"} err="failed to get container status \"a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f\": rpc error: code = NotFound desc = could not find container \"a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f\": container with ID starting with a83966b54990a1a9502c2e8ed50e6057a6c980c69234e92d402130adef2ac18f not found: ID does not exist"
Sep 17 18:41:07.786667 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.786632    2577 scope.go:117] "RemoveContainer" containerID="721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49"
Sep 17 18:41:07.786948 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:41:07.786896    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49\": container with ID starting with 721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49 not found: ID does not exist" containerID="721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49"
Sep 17 18:41:07.787010 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.786956    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49"} err="failed to get container status \"721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49\": rpc error: code = NotFound desc = could not find container \"721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49\": container with ID starting with 721c6e866916fad23ea8f78693b324bcb8ed2f6f2fa9aab565bb087fb9ce9c49 not found: ID does not exist"
Sep 17 18:41:07.796283 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.796248    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"]
Sep 17 18:41:07.799367 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:07.799339    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zs5j6"]
Sep 17 18:41:09.238507 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:09.238431    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" path="/var/lib/kubelet/pods/69a047e7-ff7a-4535-9caf-aabe5e2ae9fb/volumes"
Sep 17 18:41:11.780826 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:11.780790    2577 generic.go:356] "Generic (PLEG): container finished" podID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerID="2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801" exitCode=0
Sep 17 18:41:11.781317 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:11.780873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" event={"ID":"afce11c7-5af4-47d4-9040-f6ac4273712d","Type":"ContainerDied","Data":"2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801"}
Sep 17 18:41:12.789269 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:12.789232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" event={"ID":"afce11c7-5af4-47d4-9040-f6ac4273712d","Type":"ContainerStarted","Data":"6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32"}
Sep 17 18:41:12.789731 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:12.789291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" event={"ID":"afce11c7-5af4-47d4-9040-f6ac4273712d","Type":"ContainerStarted","Data":"aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0"}
Sep 17 18:41:12.789731 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:12.789574    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:12.789731 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:12.789718    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:12.791153 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:12.791129    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:41:12.808876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:12.808829    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podStartSLOduration=2.795240345 podStartE2EDuration="6.808812013s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.013571668s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:12.808306857 +0000 UTC m=+707.854709780" watchObservedRunningTime="2026-09-17 18:41:12.808812013 +0000 UTC m=+707.855214916"
Sep 17 18:41:13.794829 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:13.794786    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:41:18.799782 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:18.799750    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:41:18.800288 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:18.800262    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:41:28.800590 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:28.800548    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:41:38.800918 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:38.800863    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:41:48.800617 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:48.800573    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:41:58.800422 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:41:58.800379    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:42:08.800386 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:08.800345    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:42:18.800384 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:18.800334    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:42:19.233918 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:19.233859    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:42:29.234267 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:29.234213    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:42:39.238457 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:39.238381    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:42:47.523173 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:47.523138    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"]
Sep 17 18:42:47.523714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:47.523444    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" containerID="cri-o://aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0" gracePeriod=30
Sep 17 18:42:47.523714 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:47.523514    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kube-rbac-proxy" containerID="cri-o://6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32" gracePeriod=30
Sep 17 18:42:48.242343 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:48.242304    2577 generic.go:356] "Generic (PLEG): container finished" podID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerID="6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32" exitCode=2
Sep 17 18:42:48.242519 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:48.242375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" event={"ID":"afce11c7-5af4-47d4-9040-f6ac4273712d","Type":"ContainerDied","Data":"6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32"}
Sep 17 18:42:48.795185 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:48.795134    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 18:42:49.234381 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:49.234338    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:42:52.711796 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.711761    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"]
Sep 17 18:42:52.712167 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.712140    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container"
Sep 17 18:42:52.712167 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.712151    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container"
Sep 17 18:42:52.712167 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.712168    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="storage-initializer"
Sep 17 18:42:52.712280 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.712174    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="storage-initializer"
Sep 17 18:42:52.712280 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.712190    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kube-rbac-proxy"
Sep 17 18:42:52.712280 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.712196    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kube-rbac-proxy"
Sep 17 18:42:52.712280 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.712274    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kube-rbac-proxy"
Sep 17 18:42:52.712413 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.712282    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a047e7-ff7a-4535-9caf-aabe5e2ae9fb" containerName="kserve-container"
Sep 17 18:42:52.715776 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.715755    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.718180 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.718153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 18:42:52.718399 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.718385    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 18:42:52.726555 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.726523    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"]
Sep 17 18:42:52.814078 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.814045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.814257 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.814101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.814257 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.814142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.814257 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.814235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np8pl\" (UniqueName: \"kubernetes.io/projected/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-api-access-np8pl\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.814398 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.814299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.873946 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.873921    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:42:52.915218 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915184    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-cabundle-cert\") pod \"afce11c7-5af4-47d4-9040-f6ac4273712d\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") "
Sep 17 18:42:52.915469 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afce11c7-5af4-47d4-9040-f6ac4273712d-kserve-provision-location\") pod \"afce11c7-5af4-47d4-9040-f6ac4273712d\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") "
Sep 17 18:42:52.915469 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915255    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrjvc\" (UniqueName: \"kubernetes.io/projected/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-api-access-lrjvc\") pod \"afce11c7-5af4-47d4-9040-f6ac4273712d\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") "
Sep 17 18:42:52.915469 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915290    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afce11c7-5af4-47d4-9040-f6ac4273712d-proxy-tls\") pod \"afce11c7-5af4-47d4-9040-f6ac4273712d\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") "
Sep 17 18:42:52.915469 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-rbac-proxy-sar-config\") pod \"afce11c7-5af4-47d4-9040-f6ac4273712d\" (UID: \"afce11c7-5af4-47d4-9040-f6ac4273712d\") "
Sep 17 18:42:52.915469 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8pl\" (UniqueName: \"kubernetes.io/projected/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-api-access-np8pl\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.915729 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.915729 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915549    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afce11c7-5af4-47d4-9040-f6ac4273712d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afce11c7-5af4-47d4-9040-f6ac4273712d" (UID: "afce11c7-5af4-47d4-9040-f6ac4273712d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:52.915729 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afce11c7-5af4-47d4-9040-f6ac4273712d" (UID: "afce11c7-5af4-47d4-9040-f6ac4273712d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:52.915882 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915767    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afce11c7-5af4-47d4-9040-f6ac4273712d" (UID: "afce11c7-5af4-47d4-9040-f6ac4273712d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:52.915882 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.915882 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.915882 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.916108 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915960    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:42:52.916108 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915975    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afce11c7-5af4-47d4-9040-f6ac4273712d-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:42:52.916108 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.915988    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afce11c7-5af4-47d4-9040-f6ac4273712d-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:42:52.916108 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:42:52.916011    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 17 18:42:52.916108 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:42:52.916065    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-proxy-tls podName:cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f nodeName:}" failed. No retries permitted until 2026-09-17 18:42:53.416047867 +0000 UTC m=+808.462450749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" (UID: "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 17 18:42:52.916674 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.916281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.916674 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.916306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.916674 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.916342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:52.917534 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.917507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afce11c7-5af4-47d4-9040-f6ac4273712d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afce11c7-5af4-47d4-9040-f6ac4273712d" (UID: "afce11c7-5af4-47d4-9040-f6ac4273712d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:52.917728 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.917705    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-api-access-lrjvc" (OuterVolumeSpecName: "kube-api-access-lrjvc") pod "afce11c7-5af4-47d4-9040-f6ac4273712d" (UID: "afce11c7-5af4-47d4-9040-f6ac4273712d"). InnerVolumeSpecName "kube-api-access-lrjvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:52.924038 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:52.924015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8pl\" (UniqueName: \"kubernetes.io/projected/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-api-access-np8pl\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:53.017127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.017037    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrjvc\" (UniqueName: \"kubernetes.io/projected/afce11c7-5af4-47d4-9040-f6ac4273712d-kube-api-access-lrjvc\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:42:53.017127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.017075    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afce11c7-5af4-47d4-9040-f6ac4273712d-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:42:53.266214 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.266179    2577 generic.go:356] "Generic (PLEG): container finished" podID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerID="aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0" exitCode=0
Sep 17 18:42:53.266406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.266228    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"
Sep 17 18:42:53.266406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.266262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" event={"ID":"afce11c7-5af4-47d4-9040-f6ac4273712d","Type":"ContainerDied","Data":"aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0"}
Sep 17 18:42:53.266406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.266342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9" event={"ID":"afce11c7-5af4-47d4-9040-f6ac4273712d","Type":"ContainerDied","Data":"5b43ab71332a93eaac4530df40767231fd571323d06256209e08446fadb8455d"}
Sep 17 18:42:53.266406 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.266369    2577 scope.go:117] "RemoveContainer" containerID="6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32"
Sep 17 18:42:53.274932 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.274897    2577 scope.go:117] "RemoveContainer" containerID="aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0"
Sep 17 18:42:53.283058 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.283039    2577 scope.go:117] "RemoveContainer" containerID="2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801"
Sep 17 18:42:53.284424 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.284402    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"]
Sep 17 18:42:53.288116 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.288087    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-zrfg9"]
Sep 17 18:42:53.291405 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.291389    2577 scope.go:117] "RemoveContainer" containerID="6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32"
Sep 17 18:42:53.291721 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:42:53.291699    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32\": container with ID starting with 6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32 not found: ID does not exist" containerID="6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32"
Sep 17 18:42:53.291774 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.291732    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32"} err="failed to get container status \"6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32\": rpc error: code = NotFound desc = could not find container \"6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32\": container with ID starting with 6ff9e21da3230f20d31b5c531d6d4c849ee875f0a9c0a74661a9dade620fef32 not found: ID does not exist"
Sep 17 18:42:53.291774 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.291751    2577 scope.go:117] "RemoveContainer" containerID="aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0"
Sep 17 18:42:53.292072 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:42:53.292050    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0\": container with ID starting with aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0 not found: ID does not exist" containerID="aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0"
Sep 17 18:42:53.292138 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.292074    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0"} err="failed to get container status \"aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0\": rpc error: code = NotFound desc = could not find container \"aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0\": container with ID starting with aeac07ccb637798407d3f37e31a9ea02add7817fa9df0c8aae2da4ca9de3faa0 not found: ID does not exist"
Sep 17 18:42:53.292138 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.292090    2577 scope.go:117] "RemoveContainer" containerID="2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801"
Sep 17 18:42:53.292321 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:42:53.292309    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801\": container with ID starting with 2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801 not found: ID does not exist" containerID="2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801"
Sep 17 18:42:53.292359 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.292324    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801"} err="failed to get container status \"2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801\": rpc error: code = NotFound desc = could not find container \"2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801\": container with ID starting with 2ae904f77ee142ac72859758b8cbec0241f03646fcff5c79a8cfe54c11028801 not found: ID does not exist"
Sep 17 18:42:53.420497 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.420461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:53.422861 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.422842    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:53.663372 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.663262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:53.936372 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:53.936344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"]
Sep 17 18:42:53.938653 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:42:53.938626    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd74caf5_4be8_4953_b1af_8fd4d7d7dd2f.slice/crio-2af7d10daa25afa47fe76d2374ff7b9c964b7e956ca72a5d693118463a53e5f0 WatchSource:0}: Error finding container 2af7d10daa25afa47fe76d2374ff7b9c964b7e956ca72a5d693118463a53e5f0: Status 404 returned error can't find the container with id 2af7d10daa25afa47fe76d2374ff7b9c964b7e956ca72a5d693118463a53e5f0
Sep 17 18:42:54.274707 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:54.274611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" event={"ID":"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f","Type":"ContainerStarted","Data":"8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255"}
Sep 17 18:42:54.274707 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:54.274675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" event={"ID":"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f","Type":"ContainerStarted","Data":"2af7d10daa25afa47fe76d2374ff7b9c964b7e956ca72a5d693118463a53e5f0"}
Sep 17 18:42:55.245225 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:55.245190    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" path="/var/lib/kubelet/pods/afce11c7-5af4-47d4-9040-f6ac4273712d/volumes"
Sep 17 18:42:58.300121 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:58.300083    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerID="8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255" exitCode=0
Sep 17 18:42:58.300518 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:58.300157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" event={"ID":"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f","Type":"ContainerDied","Data":"8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255"}
Sep 17 18:42:59.307677 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:59.307638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" event={"ID":"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f","Type":"ContainerStarted","Data":"8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554"}
Sep 17 18:42:59.308188 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:59.307711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" event={"ID":"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f","Type":"ContainerStarted","Data":"32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a"}
Sep 17 18:42:59.308188 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:59.307938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:42:59.330236 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:42:59.330169    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podStartSLOduration=3.486905483 podStartE2EDuration="7.330151582s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.843246099s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:59.328575877 +0000 UTC m=+814.374978779" watchObservedRunningTime="2026-09-17 18:42:59.330151582 +0000 UTC m=+814.376554483"
Sep 17 18:43:00.312934 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:00.312880    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:43:00.314418 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:00.314384    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:43:01.317748 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:01.317708    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:43:06.322162 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:06.322131    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:43:06.322624 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:06.322597    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:43:16.322889 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:16.322848    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:43:26.322709 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:26.322667    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:43:36.322684 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:36.322642    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:43:46.323448 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:46.323405    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:43:56.323301 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.323253    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:43:56.920379 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.920340    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"]
Sep 17 18:43:56.920876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.920842    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container"
Sep 17 18:43:56.920876 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.920867    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container"
Sep 17 18:43:56.921062 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.920890    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kube-rbac-proxy"
Sep 17 18:43:56.921062 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.920897    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kube-rbac-proxy"
Sep 17 18:43:56.921062 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.920928    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="storage-initializer"
Sep 17 18:43:56.921062 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.920935    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="storage-initializer"
Sep 17 18:43:56.921062 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.921000    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kserve-container"
Sep 17 18:43:56.921062 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.921011    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="afce11c7-5af4-47d4-9040-f6ac4273712d" containerName="kube-rbac-proxy"
Sep 17 18:43:56.924219 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.924204    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:56.926624 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.926606    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 18:43:56.926683 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.926609    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 18:43:56.937181 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.937153    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"]
Sep 17 18:43:56.984578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.984534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxh5m\" (UniqueName: \"kubernetes.io/projected/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-api-access-bxh5m\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:56.984758 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.984621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:56.984758 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.984654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:56.984758 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.984686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:56.984758 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:56.984712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.085545 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.085507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.085728 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.085559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.085728 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.085590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.085728 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.085612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.085728 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.085632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxh5m\" (UniqueName: \"kubernetes.io/projected/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-api-access-bxh5m\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.086147 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.086128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.086339 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.086317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.086474 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.086450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.088126 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.088107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.095187 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.095167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxh5m\" (UniqueName: \"kubernetes.io/projected/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-api-access-bxh5m\") pod \"isvc-autogluon-v1-predictor-574b5b7495-txdcl\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.235224 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.235165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:43:57.509370 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.509342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"]
Sep 17 18:43:57.511433 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:43:57.511402    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b30808_60df_40c7_b0f3_6a9d1fd6f43c.slice/crio-5a2c51e07d2e9e69605315f79492faa37b569e2d25b438ab1a4b9e18c31b3b7d WatchSource:0}: Error finding container 5a2c51e07d2e9e69605315f79492faa37b569e2d25b438ab1a4b9e18c31b3b7d: Status 404 returned error can't find the container with id 5a2c51e07d2e9e69605315f79492faa37b569e2d25b438ab1a4b9e18c31b3b7d
Sep 17 18:43:57.588804 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.588766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" event={"ID":"92b30808-60df-40c7-b0f3-6a9d1fd6f43c","Type":"ContainerStarted","Data":"c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2"}
Sep 17 18:43:57.589018 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:43:57.588828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" event={"ID":"92b30808-60df-40c7-b0f3-6a9d1fd6f43c","Type":"ContainerStarted","Data":"5a2c51e07d2e9e69605315f79492faa37b569e2d25b438ab1a4b9e18c31b3b7d"}
Sep 17 18:44:06.322850 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:06.322811    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:44:16.322848 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:16.322757    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:44:24.234127 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:24.234085    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:44:26.723712 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:26.723672    2577 generic.go:356] "Generic (PLEG): container finished" podID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerID="c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2" exitCode=0
Sep 17 18:44:26.724164 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:26.723721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" event={"ID":"92b30808-60df-40c7-b0f3-6a9d1fd6f43c","Type":"ContainerDied","Data":"c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2"}
Sep 17 18:44:26.725070 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:26.725051    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:44:33.734731 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:33.734698    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"]
Sep 17 18:44:33.735246 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:33.734978    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" containerID="cri-o://32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a" gracePeriod=30
Sep 17 18:44:33.735246 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:33.734985    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kube-rbac-proxy" containerID="cri-o://8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554" gracePeriod=30
Sep 17 18:44:34.233725 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:34.233671    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:44:34.773626 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:34.773584    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerID="8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554" exitCode=2
Sep 17 18:44:34.774012 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:34.773657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" event={"ID":"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f","Type":"ContainerDied","Data":"8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554"}
Sep 17 18:44:36.318754 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:36.318699    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 18:44:39.487634 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.487602    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:44:39.579878 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.579786    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kserve-provision-location\") pod \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") "
Sep 17 18:44:39.579878 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.579848    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np8pl\" (UniqueName: \"kubernetes.io/projected/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-api-access-np8pl\") pod \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") "
Sep 17 18:44:39.580336 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.579890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-proxy-tls\") pod \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") "
Sep 17 18:44:39.580336 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.579942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-rbac-proxy-sar-config\") pod \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") "
Sep 17 18:44:39.580336 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.580000    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-cabundle-cert\") pod \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\" (UID: \"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f\") "
Sep 17 18:44:39.580336 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.580215    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" (UID: "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:39.580336 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.580308    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:44:39.580623 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.580513    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" (UID: "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:44:39.580623 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.580585    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" (UID: "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:44:39.582303 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.582277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" (UID: "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:39.582432 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.582371    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-api-access-np8pl" (OuterVolumeSpecName: "kube-api-access-np8pl") pod "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" (UID: "cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f"). InnerVolumeSpecName "kube-api-access-np8pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:39.681330 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.681288    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:44:39.681330 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.681329    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:44:39.681538 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.681343    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:44:39.681538 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.681358    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np8pl\" (UniqueName: \"kubernetes.io/projected/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f-kube-api-access-np8pl\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:44:39.802524 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.802486    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerID="32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a" exitCode=0
Sep 17 18:44:39.802698 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.802553    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"
Sep 17 18:44:39.802698 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.802558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" event={"ID":"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f","Type":"ContainerDied","Data":"32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a"}
Sep 17 18:44:39.802698 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.802618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk" event={"ID":"cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f","Type":"ContainerDied","Data":"2af7d10daa25afa47fe76d2374ff7b9c964b7e956ca72a5d693118463a53e5f0"}
Sep 17 18:44:39.802698 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.802648    2577 scope.go:117] "RemoveContainer" containerID="8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554"
Sep 17 18:44:39.812955 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.812931    2577 scope.go:117] "RemoveContainer" containerID="32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a"
Sep 17 18:44:39.822196 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.822178    2577 scope.go:117] "RemoveContainer" containerID="8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255"
Sep 17 18:44:39.827581 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.827556    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"]
Sep 17 18:44:39.831324 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.831271    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-sx8xk"]
Sep 17 18:44:39.832729 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.832713    2577 scope.go:117] "RemoveContainer" containerID="8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554"
Sep 17 18:44:39.833061 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:44:39.833039    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554\": container with ID starting with 8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554 not found: ID does not exist" containerID="8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554"
Sep 17 18:44:39.833128 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.833069    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554"} err="failed to get container status \"8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554\": rpc error: code = NotFound desc = could not find container \"8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554\": container with ID starting with 8513ca05582bfa4ab9d4bacc8068fb60e06aa04dc08ca0dfa1a5eea9392a2554 not found: ID does not exist"
Sep 17 18:44:39.833128 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.833100    2577 scope.go:117] "RemoveContainer" containerID="32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a"
Sep 17 18:44:39.833383 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:44:39.833366    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a\": container with ID starting with 32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a not found: ID does not exist" containerID="32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a"
Sep 17 18:44:39.833449 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.833388    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a"} err="failed to get container status \"32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a\": rpc error: code = NotFound desc = could not find container \"32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a\": container with ID starting with 32ec15e9e184fd136b0860f2f6def8547aa10797f019911d22a8d78d11cfda2a not found: ID does not exist"
Sep 17 18:44:39.833449 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.833403    2577 scope.go:117] "RemoveContainer" containerID="8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255"
Sep 17 18:44:39.833683 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:44:39.833662    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255\": container with ID starting with 8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255 not found: ID does not exist" containerID="8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255"
Sep 17 18:44:39.833741 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:39.833688    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255"} err="failed to get container status \"8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255\": rpc error: code = NotFound desc = could not find container \"8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255\": container with ID starting with 8e2774230a32055562f19617d80a5f726dbe6f711d8e0a13cdc76144c0d93255 not found: ID does not exist"
Sep 17 18:44:41.240157 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:44:41.240121    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" path="/var/lib/kubelet/pods/cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f/volumes"
Sep 17 18:46:25.364284 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:25.364193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" event={"ID":"92b30808-60df-40c7-b0f3-6a9d1fd6f43c","Type":"ContainerStarted","Data":"e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703"}
Sep 17 18:46:25.364284 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:25.364265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" event={"ID":"92b30808-60df-40c7-b0f3-6a9d1fd6f43c","Type":"ContainerStarted","Data":"adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797"}
Sep 17 18:46:25.364691 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:25.364396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:46:25.399487 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:25.399432    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" podStartSLOduration=2.132976064 podStartE2EDuration="2m29.399403791s" totalImagesPullingTime="1m58.325756158s" totalInitContainerRuntime="28.940671569s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:25.396833649 +0000 UTC m=+1020.443236576" watchObservedRunningTime="2026-09-17 18:46:25.399403791 +0000 UTC m=+1020.445806692"
Sep 17 18:46:26.370269 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:26.370210    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:46:26.371727 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:26.371700    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:46:27.374318 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:27.374274    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:46:32.378424 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:32.378394    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:46:32.379085 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:32.379055    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:46:42.380324 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:42.380289    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:46:53.309416 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:53.309379    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"]
Sep 17 18:46:53.309843 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:53.309694    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kserve-container" containerID="cri-o://adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797" gracePeriod=30
Sep 17 18:46:53.309843 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:53.309715    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kube-rbac-proxy" containerID="cri-o://e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703" gracePeriod=30
Sep 17 18:46:53.497502 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:53.497465    2577 generic.go:356] "Generic (PLEG): container finished" podID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerID="e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703" exitCode=2
Sep 17 18:46:53.497677 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:53.497535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" event={"ID":"92b30808-60df-40c7-b0f3-6a9d1fd6f43c","Type":"ContainerDied","Data":"e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703"}
Sep 17 18:46:54.894310 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.894271    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:46:54.960259 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.960223    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-cabundle-cert\") pod \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") "
Sep 17 18:46:54.960430 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.960293    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-rbac-proxy-sar-config\") pod \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") "
Sep 17 18:46:54.960430 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.960322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxh5m\" (UniqueName: \"kubernetes.io/projected/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-api-access-bxh5m\") pod \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") "
Sep 17 18:46:54.960430 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.960367    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kserve-provision-location\") pod \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") "
Sep 17 18:46:54.960574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.960555    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-proxy-tls\") pod \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\" (UID: \"92b30808-60df-40c7-b0f3-6a9d1fd6f43c\") "
Sep 17 18:46:54.960685 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.960658    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92b30808-60df-40c7-b0f3-6a9d1fd6f43c" (UID: "92b30808-60df-40c7-b0f3-6a9d1fd6f43c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:54.960864 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.960744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92b30808-60df-40c7-b0f3-6a9d1fd6f43c" (UID: "92b30808-60df-40c7-b0f3-6a9d1fd6f43c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:54.960864 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.960857    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:46:54.961022 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.960875    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:46:54.962478 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.962454    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-api-access-bxh5m" (OuterVolumeSpecName: "kube-api-access-bxh5m") pod "92b30808-60df-40c7-b0f3-6a9d1fd6f43c" (UID: "92b30808-60df-40c7-b0f3-6a9d1fd6f43c"). InnerVolumeSpecName "kube-api-access-bxh5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:54.962575 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.962560    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92b30808-60df-40c7-b0f3-6a9d1fd6f43c" (UID: "92b30808-60df-40c7-b0f3-6a9d1fd6f43c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:54.965045 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:54.965021    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92b30808-60df-40c7-b0f3-6a9d1fd6f43c" (UID: "92b30808-60df-40c7-b0f3-6a9d1fd6f43c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:55.061439 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.061402    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:46:55.061439 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.061435    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxh5m\" (UniqueName: \"kubernetes.io/projected/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kube-api-access-bxh5m\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:46:55.061439 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.061444    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92b30808-60df-40c7-b0f3-6a9d1fd6f43c-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:46:55.509969 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.509873    2577 generic.go:356] "Generic (PLEG): container finished" podID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerID="adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797" exitCode=0
Sep 17 18:46:55.510129 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.509968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" event={"ID":"92b30808-60df-40c7-b0f3-6a9d1fd6f43c","Type":"ContainerDied","Data":"adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797"}
Sep 17 18:46:55.510129 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.510006    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"
Sep 17 18:46:55.510129 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.510033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl" event={"ID":"92b30808-60df-40c7-b0f3-6a9d1fd6f43c","Type":"ContainerDied","Data":"5a2c51e07d2e9e69605315f79492faa37b569e2d25b438ab1a4b9e18c31b3b7d"}
Sep 17 18:46:55.510129 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.510070    2577 scope.go:117] "RemoveContainer" containerID="e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703"
Sep 17 18:46:55.518935 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.518884    2577 scope.go:117] "RemoveContainer" containerID="adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797"
Sep 17 18:46:55.527213 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.527191    2577 scope.go:117] "RemoveContainer" containerID="c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2"
Sep 17 18:46:55.532243 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.532219    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"]
Sep 17 18:46:55.535897 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.535874    2577 scope.go:117] "RemoveContainer" containerID="e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703"
Sep 17 18:46:55.536260 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:46:55.536241    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703\": container with ID starting with e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703 not found: ID does not exist" containerID="e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703"
Sep 17 18:46:55.536314 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.536270    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703"} err="failed to get container status \"e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703\": rpc error: code = NotFound desc = could not find container \"e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703\": container with ID starting with e5ae8b44ebda4bf191009f61602686806aa71d40e4873c8cb794891dbe410703 not found: ID does not exist"
Sep 17 18:46:55.536314 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.536302    2577 scope.go:117] "RemoveContainer" containerID="adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797"
Sep 17 18:46:55.536602 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:46:55.536585    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797\": container with ID starting with adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797 not found: ID does not exist" containerID="adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797"
Sep 17 18:46:55.536644 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.536608    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797"} err="failed to get container status \"adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797\": rpc error: code = NotFound desc = could not find container \"adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797\": container with ID starting with adcd0dd261242298392c5c4fc34c321e2b7484f26d3bc922004c868c32861797 not found: ID does not exist"
Sep 17 18:46:55.536644 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.536624    2577 scope.go:117] "RemoveContainer" containerID="c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2"
Sep 17 18:46:55.536889 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:46:55.536871    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2\": container with ID starting with c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2 not found: ID does not exist" containerID="c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2"
Sep 17 18:46:55.536989 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.536889    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2"} err="failed to get container status \"c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2\": rpc error: code = NotFound desc = could not find container \"c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2\": container with ID starting with c8acc56de0016dcc23460a44e77e27b0dc13b17b65091b4a0582219fdf31afb2 not found: ID does not exist"
Sep 17 18:46:55.537227 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:55.537208    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-txdcl"]
Sep 17 18:46:57.239678 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:57.239643    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" path="/var/lib/kubelet/pods/92b30808-60df-40c7-b0f3-6a9d1fd6f43c/volumes"
Sep 17 18:46:58.687194 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687160    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"]
Sep 17 18:46:58.687622 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687557    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kube-rbac-proxy"
Sep 17 18:46:58.687622 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687569    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kube-rbac-proxy"
Sep 17 18:46:58.687622 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687578    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container"
Sep 17 18:46:58.687622 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687584    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container"
Sep 17 18:46:58.687622 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687591    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="storage-initializer"
Sep 17 18:46:58.687622 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687596    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="storage-initializer"
Sep 17 18:46:58.687622 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687606    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kube-rbac-proxy"
Sep 17 18:46:58.687622 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687612    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kube-rbac-proxy"
Sep 17 18:46:58.687622 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687619    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kserve-container"
Sep 17 18:46:58.687622 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687624    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kserve-container"
Sep 17 18:46:58.688001 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687641    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="storage-initializer"
Sep 17 18:46:58.688001 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687647    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="storage-initializer"
Sep 17 18:46:58.688001 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687698    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kserve-container"
Sep 17 18:46:58.688001 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687707    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kube-rbac-proxy"
Sep 17 18:46:58.688001 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687713    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd74caf5-4be8-4953-b1af-8fd4d7d7dd2f" containerName="kube-rbac-proxy"
Sep 17 18:46:58.688001 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.687720    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b30808-60df-40c7-b0f3-6a9d1fd6f43c" containerName="kserve-container"
Sep 17 18:46:58.710548 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.710518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.715101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.715079    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:46:58.715101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.715080    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:46:58.715332 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.715086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 18:46:58.715332 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.715161    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:46:58.716317 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.716292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 18:46:58.716317 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.716308    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:46:58.716531 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.716381    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:46:58.726079 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.726059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"]
Sep 17 18:46:58.794863 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.794830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.795157 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.794893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.795157 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.794976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv69j\" (UniqueName: \"kubernetes.io/projected/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-api-access-qv69j\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.795157 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.795045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.795157 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.795078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.895882 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.895847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.896106 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.895925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.896106 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.895960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv69j\" (UniqueName: \"kubernetes.io/projected/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-api-access-qv69j\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.896106 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.895988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.896106 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.896003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.896490 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.896461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.896686 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.896666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.896830 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.896811    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.898379 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.898361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:58.916794 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:58.916765    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv69j\" (UniqueName: \"kubernetes.io/projected/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-api-access-qv69j\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-d685q\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:59.021041 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:59.020939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:46:59.338276 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:59.338250    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"]
Sep 17 18:46:59.340772 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:46:59.340745    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1856b3_ce91_485e_a6eb_ad9156ee3a40.slice/crio-a305c874a705a1c6eb8260a7cb6487a17734c35e1c27908e2f84501229bcf8aa WatchSource:0}: Error finding container a305c874a705a1c6eb8260a7cb6487a17734c35e1c27908e2f84501229bcf8aa: Status 404 returned error can't find the container with id a305c874a705a1c6eb8260a7cb6487a17734c35e1c27908e2f84501229bcf8aa
Sep 17 18:46:59.533670 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:59.533627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" event={"ID":"3c1856b3-ce91-485e-a6eb-ad9156ee3a40","Type":"ContainerStarted","Data":"f3946a7ba40b84a6607911abc64cb85880be613db043867d8305d667a79e66b3"}
Sep 17 18:46:59.533838 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:46:59.533690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" event={"ID":"3c1856b3-ce91-485e-a6eb-ad9156ee3a40","Type":"ContainerStarted","Data":"a305c874a705a1c6eb8260a7cb6487a17734c35e1c27908e2f84501229bcf8aa"}
Sep 17 18:47:04.560987 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:47:04.560890    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerID="f3946a7ba40b84a6607911abc64cb85880be613db043867d8305d667a79e66b3" exitCode=0
Sep 17 18:47:04.560987 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:47:04.560933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" event={"ID":"3c1856b3-ce91-485e-a6eb-ad9156ee3a40","Type":"ContainerDied","Data":"f3946a7ba40b84a6607911abc64cb85880be613db043867d8305d667a79e66b3"}
Sep 17 18:47:05.568192 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:47:05.568157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" event={"ID":"3c1856b3-ce91-485e-a6eb-ad9156ee3a40","Type":"ContainerStarted","Data":"5d0d414f7d7b426327368ac6b873a3066d3e4a124aa2b53be3696c3ea90f9f75"}
Sep 17 18:47:05.568618 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:47:05.568219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" event={"ID":"3c1856b3-ce91-485e-a6eb-ad9156ee3a40","Type":"ContainerStarted","Data":"de6bdf95cd6bc01525c9f6a218a73a89cda3c10ca2bbf72b6d004a84e4500c62"}
Sep 17 18:47:05.568618 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:47:05.568536    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:47:05.568618 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:47:05.568582    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:47:05.589195 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:47:05.589142    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" podStartSLOduration=2.778215184 podStartE2EDuration="7.589124849s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.810909665s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:05.586610381 +0000 UTC m=+1060.633013285" watchObservedRunningTime="2026-09-17 18:47:05.589124849 +0000 UTC m=+1060.635527750"
Sep 17 18:47:11.579978 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:47:11.579878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:48:34.266601 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:34.266571    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:48:44.554377 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:44.554344    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"]
Sep 17 18:48:44.554795 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:44.554585    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerName="kserve-container" containerID="cri-o://de6bdf95cd6bc01525c9f6a218a73a89cda3c10ca2bbf72b6d004a84e4500c62" gracePeriod=30
Sep 17 18:48:44.554795 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:44.554625    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerName="kube-rbac-proxy" containerID="cri-o://5d0d414f7d7b426327368ac6b873a3066d3e4a124aa2b53be3696c3ea90f9f75" gracePeriod=30
Sep 17 18:48:45.036040 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:45.035990    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerID="5d0d414f7d7b426327368ac6b873a3066d3e4a124aa2b53be3696c3ea90f9f75" exitCode=2
Sep 17 18:48:45.036217 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:45.036039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" event={"ID":"3c1856b3-ce91-485e-a6eb-ad9156ee3a40","Type":"ContainerDied","Data":"5d0d414f7d7b426327368ac6b873a3066d3e4a124aa2b53be3696c3ea90f9f75"}
Sep 17 18:48:46.045726 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.045697    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerID="de6bdf95cd6bc01525c9f6a218a73a89cda3c10ca2bbf72b6d004a84e4500c62" exitCode=0
Sep 17 18:48:46.046055 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.045753    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" event={"ID":"3c1856b3-ce91-485e-a6eb-ad9156ee3a40","Type":"ContainerDied","Data":"de6bdf95cd6bc01525c9f6a218a73a89cda3c10ca2bbf72b6d004a84e4500c62"}
Sep 17 18:48:46.091782 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.091762    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:48:46.162476 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.162417    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-cabundle-cert\") pod \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") "
Sep 17 18:48:46.162476 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.162450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kserve-provision-location\") pod \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") "
Sep 17 18:48:46.162632 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.162500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv69j\" (UniqueName: \"kubernetes.io/projected/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-api-access-qv69j\") pod \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") "
Sep 17 18:48:46.162632 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.162608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-rbac-proxy-sar-config\") pod \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") "
Sep 17 18:48:46.162715 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.162669    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-proxy-tls\") pod \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\" (UID: \"3c1856b3-ce91-485e-a6eb-ad9156ee3a40\") "
Sep 17 18:48:46.162857 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.162830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c1856b3-ce91-485e-a6eb-ad9156ee3a40" (UID: "3c1856b3-ce91-485e-a6eb-ad9156ee3a40"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:48:46.163006 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.162987    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:48:46.163006 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.162980    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c1856b3-ce91-485e-a6eb-ad9156ee3a40" (UID: "3c1856b3-ce91-485e-a6eb-ad9156ee3a40"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:48:46.164543 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.164520    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-api-access-qv69j" (OuterVolumeSpecName: "kube-api-access-qv69j") pod "3c1856b3-ce91-485e-a6eb-ad9156ee3a40" (UID: "3c1856b3-ce91-485e-a6eb-ad9156ee3a40"). InnerVolumeSpecName "kube-api-access-qv69j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:48:46.164672 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.164649    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c1856b3-ce91-485e-a6eb-ad9156ee3a40" (UID: "3c1856b3-ce91-485e-a6eb-ad9156ee3a40"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:48:46.166849 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.166825    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c1856b3-ce91-485e-a6eb-ad9156ee3a40" (UID: "3c1856b3-ce91-485e-a6eb-ad9156ee3a40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:48:46.263613 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.263590    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:48:46.263613 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.263614    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv69j\" (UniqueName: \"kubernetes.io/projected/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-api-access-qv69j\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:48:46.263735 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.263698    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:48:46.263735 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:46.263713    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c1856b3-ce91-485e-a6eb-ad9156ee3a40-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:48:47.052395 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:47.052371    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"
Sep 17 18:48:47.052747 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:47.052370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q" event={"ID":"3c1856b3-ce91-485e-a6eb-ad9156ee3a40","Type":"ContainerDied","Data":"a305c874a705a1c6eb8260a7cb6487a17734c35e1c27908e2f84501229bcf8aa"}
Sep 17 18:48:47.052747 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:47.052504    2577 scope.go:117] "RemoveContainer" containerID="5d0d414f7d7b426327368ac6b873a3066d3e4a124aa2b53be3696c3ea90f9f75"
Sep 17 18:48:47.060978 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:47.060964    2577 scope.go:117] "RemoveContainer" containerID="de6bdf95cd6bc01525c9f6a218a73a89cda3c10ca2bbf72b6d004a84e4500c62"
Sep 17 18:48:47.068553 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:47.068528    2577 scope.go:117] "RemoveContainer" containerID="f3946a7ba40b84a6607911abc64cb85880be613db043867d8305d667a79e66b3"
Sep 17 18:48:47.078851 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:47.078828    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"]
Sep 17 18:48:47.083074 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:47.083053    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-d685q"]
Sep 17 18:48:47.238133 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:47.238107    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" path="/var/lib/kubelet/pods/3c1856b3-ce91-485e-a6eb-ad9156ee3a40/volumes"
Sep 17 18:48:49.772924 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.772867    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"]
Sep 17 18:48:49.773291 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.773245    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerName="storage-initializer"
Sep 17 18:48:49.773291 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.773259    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerName="storage-initializer"
Sep 17 18:48:49.773291 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.773276    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerName="kserve-container"
Sep 17 18:48:49.773291 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.773282    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerName="kserve-container"
Sep 17 18:48:49.773291 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.773291    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerName="kube-rbac-proxy"
Sep 17 18:48:49.773476 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.773297    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerName="kube-rbac-proxy"
Sep 17 18:48:49.773476 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.773356    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerName="kube-rbac-proxy"
Sep 17 18:48:49.773476 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.773364    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1856b3-ce91-485e-a6eb-ad9156ee3a40" containerName="kserve-container"
Sep 17 18:48:49.776581 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.776564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.779808 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.779782    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:48:49.779961 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.779784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:48:49.779961 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.779844    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:48:49.780455 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.780434    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 18:48:49.780455 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.780444    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:48:49.780627 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.780562    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 18:48:49.780989 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.780965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 18:48:49.786856 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.786832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"]
Sep 17 18:48:49.897984 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.897940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.898185 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.898009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.898185 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.898039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbjkp\" (UniqueName: \"kubernetes.io/projected/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-api-access-wbjkp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.898185 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.898064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.898353 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.898184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.998605 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.998573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.998725 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.998610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.998725 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.998648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.998725 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.998666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbjkp\" (UniqueName: \"kubernetes.io/projected/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-api-access-wbjkp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.998725 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.998683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.998965 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:48:49.998813    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 18:48:49.998965 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:48:49.998927    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-proxy-tls podName:dc1bd3a3-75c7-4061-b19c-a372c554d5ea nodeName:}" failed. No retries permitted until 2026-09-17 18:48:50.498883149 +0000 UTC m=+1165.545286043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" (UID: "dc1bd3a3-75c7-4061-b19c-a372c554d5ea") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 18:48:49.999119 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.999099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.999330 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.999313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:49.999368 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:49.999331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:50.007642 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:50.007625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbjkp\" (UniqueName: \"kubernetes.io/projected/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-api-access-wbjkp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:50.504213 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:50.504175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:50.506612 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:50.506582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:50.688219 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:50.688178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:50.957041 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:50.957016    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"]
Sep 17 18:48:50.959136 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:48:50.959103    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1bd3a3_75c7_4061_b19c_a372c554d5ea.slice/crio-2a53b8b11f07c1a5fbd42a2122ea820ff9672a0f62a09e114206f87747fb8e31 WatchSource:0}: Error finding container 2a53b8b11f07c1a5fbd42a2122ea820ff9672a0f62a09e114206f87747fb8e31: Status 404 returned error can't find the container with id 2a53b8b11f07c1a5fbd42a2122ea820ff9672a0f62a09e114206f87747fb8e31
Sep 17 18:48:51.073143 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:51.073102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" event={"ID":"dc1bd3a3-75c7-4061-b19c-a372c554d5ea","Type":"ContainerStarted","Data":"6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8"}
Sep 17 18:48:51.073331 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:51.073161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" event={"ID":"dc1bd3a3-75c7-4061-b19c-a372c554d5ea","Type":"ContainerStarted","Data":"2a53b8b11f07c1a5fbd42a2122ea820ff9672a0f62a09e114206f87747fb8e31"}
Sep 17 18:48:56.096845 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:56.096809    2577 generic.go:356] "Generic (PLEG): container finished" podID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerID="6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8" exitCode=0
Sep 17 18:48:56.097235 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:56.096868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" event={"ID":"dc1bd3a3-75c7-4061-b19c-a372c554d5ea","Type":"ContainerDied","Data":"6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8"}
Sep 17 18:48:57.104252 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:57.104206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" event={"ID":"dc1bd3a3-75c7-4061-b19c-a372c554d5ea","Type":"ContainerStarted","Data":"b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708"}
Sep 17 18:48:57.104627 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:57.104295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" event={"ID":"dc1bd3a3-75c7-4061-b19c-a372c554d5ea","Type":"ContainerStarted","Data":"121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323"}
Sep 17 18:48:57.104627 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:57.104503    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:48:57.126454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:57.126409    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" podStartSLOduration=3.215743048 podStartE2EDuration="8.126395031s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.910651983s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:57.125297185 +0000 UTC m=+1172.171700108" watchObservedRunningTime="2026-09-17 18:48:57.126395031 +0000 UTC m=+1172.172797933"
Sep 17 18:48:58.110231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:48:58.110192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:49:04.122657 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:49:04.122629    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:50:34.268185 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:34.268148    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:50:45.275103 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:45.275055    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:50:46.371368 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:46.371334    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"]
Sep 17 18:50:46.371770 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:46.371586    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="kserve-container" containerID="cri-o://121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323" gracePeriod=30
Sep 17 18:50:46.371770 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:46.371642    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="kube-rbac-proxy" containerID="cri-o://b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708" gracePeriod=30
Sep 17 18:50:46.623865 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:46.623771    2577 generic.go:356] "Generic (PLEG): container finished" podID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerID="b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708" exitCode=2
Sep 17 18:50:46.623865 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:46.623848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" event={"ID":"dc1bd3a3-75c7-4061-b19c-a372c554d5ea","Type":"ContainerDied","Data":"b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708"}
Sep 17 18:50:47.913537 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:47.913506    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:50:48.006890 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.006776    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbjkp\" (UniqueName: \"kubernetes.io/projected/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-api-access-wbjkp\") pod \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") "
Sep 17 18:50:48.006890 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.006838    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-rbac-proxy-sar-config\") pod \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") "
Sep 17 18:50:48.006890 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.006880    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-proxy-tls\") pod \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") "
Sep 17 18:50:48.007207 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.006991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kserve-provision-location\") pod \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") "
Sep 17 18:50:48.007207 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.007033    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-cabundle-cert\") pod \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\" (UID: \"dc1bd3a3-75c7-4061-b19c-a372c554d5ea\") "
Sep 17 18:50:48.007304 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.007241    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc1bd3a3-75c7-4061-b19c-a372c554d5ea" (UID: "dc1bd3a3-75c7-4061-b19c-a372c554d5ea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:50:48.007454 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.007433    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc1bd3a3-75c7-4061-b19c-a372c554d5ea" (UID: "dc1bd3a3-75c7-4061-b19c-a372c554d5ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:50:48.009030 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.009007    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc1bd3a3-75c7-4061-b19c-a372c554d5ea" (UID: "dc1bd3a3-75c7-4061-b19c-a372c554d5ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:50:48.009145 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.009132    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-api-access-wbjkp" (OuterVolumeSpecName: "kube-api-access-wbjkp") pod "dc1bd3a3-75c7-4061-b19c-a372c554d5ea" (UID: "dc1bd3a3-75c7-4061-b19c-a372c554d5ea"). InnerVolumeSpecName "kube-api-access-wbjkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:50:48.011148 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.011127    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc1bd3a3-75c7-4061-b19c-a372c554d5ea" (UID: "dc1bd3a3-75c7-4061-b19c-a372c554d5ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:50:48.107778 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.107730    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:50:48.107778 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.107772    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbjkp\" (UniqueName: \"kubernetes.io/projected/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-api-access-wbjkp\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:50:48.107778 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.107783    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:50:48.107778 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.107791    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:50:48.108080 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.107800    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd3a3-75c7-4061-b19c-a372c554d5ea-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:50:48.635734 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.635697    2577 generic.go:356] "Generic (PLEG): container finished" podID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerID="121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323" exitCode=0
Sep 17 18:50:48.635958 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.635774    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"
Sep 17 18:50:48.635958 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.635772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" event={"ID":"dc1bd3a3-75c7-4061-b19c-a372c554d5ea","Type":"ContainerDied","Data":"121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323"}
Sep 17 18:50:48.635958 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.635938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp" event={"ID":"dc1bd3a3-75c7-4061-b19c-a372c554d5ea","Type":"ContainerDied","Data":"2a53b8b11f07c1a5fbd42a2122ea820ff9672a0f62a09e114206f87747fb8e31"}
Sep 17 18:50:48.636124 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.635967    2577 scope.go:117] "RemoveContainer" containerID="b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708"
Sep 17 18:50:48.644721 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.644700    2577 scope.go:117] "RemoveContainer" containerID="121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323"
Sep 17 18:50:48.652566 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.652547    2577 scope.go:117] "RemoveContainer" containerID="6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8"
Sep 17 18:50:48.659658 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.659633    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"]
Sep 17 18:50:48.661419 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.661402    2577 scope.go:117] "RemoveContainer" containerID="b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708"
Sep 17 18:50:48.662402 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:50:48.662220    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708\": container with ID starting with b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708 not found: ID does not exist" containerID="b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708"
Sep 17 18:50:48.662402 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.662268    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708"} err="failed to get container status \"b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708\": rpc error: code = NotFound desc = could not find container \"b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708\": container with ID starting with b1f369154cca1f2abe228a6bb5cbf6f2122f73f326cb5f6073bb69321425b708 not found: ID does not exist"
Sep 17 18:50:48.662402 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.662295    2577 scope.go:117] "RemoveContainer" containerID="121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323"
Sep 17 18:50:48.662937 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:50:48.662601    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323\": container with ID starting with 121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323 not found: ID does not exist" containerID="121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323"
Sep 17 18:50:48.662937 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.662637    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323"} err="failed to get container status \"121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323\": rpc error: code = NotFound desc = could not find container \"121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323\": container with ID starting with 121dfd9bd23459ef607a5609673d4894553fab5a83d8e09a8b779af9c2d7c323 not found: ID does not exist"
Sep 17 18:50:48.662937 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.662656    2577 scope.go:117] "RemoveContainer" containerID="6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8"
Sep 17 18:50:48.663097 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:50:48.662979    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8\": container with ID starting with 6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8 not found: ID does not exist" containerID="6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8"
Sep 17 18:50:48.663097 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.663005    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8"} err="failed to get container status \"6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8\": rpc error: code = NotFound desc = could not find container \"6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8\": container with ID starting with 6aedc337dcc346566c345daa56f0fad4d1de6719572216c6fabf3d837c5b44a8 not found: ID does not exist"
Sep 17 18:50:48.664981 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:48.664960    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8wkvp"]
Sep 17 18:50:49.238018 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:49.237986    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" path="/var/lib/kubelet/pods/dc1bd3a3-75c7-4061-b19c-a372c554d5ea/volumes"
Sep 17 18:50:51.651587 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.651554    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"]
Sep 17 18:50:51.651971 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.651918    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="storage-initializer"
Sep 17 18:50:51.651971 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.651930    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="storage-initializer"
Sep 17 18:50:51.651971 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.651950    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="kserve-container"
Sep 17 18:50:51.651971 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.651956    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="kserve-container"
Sep 17 18:50:51.651971 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.651968    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="kube-rbac-proxy"
Sep 17 18:50:51.651971 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.651973    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="kube-rbac-proxy"
Sep 17 18:50:51.652173 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.652024    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="kserve-container"
Sep 17 18:50:51.652173 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.652032    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc1bd3a3-75c7-4061-b19c-a372c554d5ea" containerName="kube-rbac-proxy"
Sep 17 18:50:51.656617 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.656597    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.662749 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.662728    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:50:51.662869 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.662795    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:50:51.663482 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.663460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:50:51.663696 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.663515    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 18:50:51.663696 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.663532    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 18:50:51.663696 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.663618    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 18:50:51.663835 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.663793    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:50:51.674163 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.674136    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"]
Sep 17 18:50:51.745008 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.744964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.745008 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.745004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90100a76-0fef-4430-b94f-05171ad876b1-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.745248 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.745049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcxw\" (UniqueName: \"kubernetes.io/projected/90100a76-0fef-4430-b94f-05171ad876b1-kube-api-access-bbcxw\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.745248 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.745115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90100a76-0fef-4430-b94f-05171ad876b1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.745248 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.745146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.846153 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.846122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.846153 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.846157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90100a76-0fef-4430-b94f-05171ad876b1-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.846382 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.846198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcxw\" (UniqueName: \"kubernetes.io/projected/90100a76-0fef-4430-b94f-05171ad876b1-kube-api-access-bbcxw\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.846382 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.846220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90100a76-0fef-4430-b94f-05171ad876b1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.846382 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.846237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.846666 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.846644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90100a76-0fef-4430-b94f-05171ad876b1-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.846795 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.846776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.846871 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.846854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.848704 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.848680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90100a76-0fef-4430-b94f-05171ad876b1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.856130 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.856098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcxw\" (UniqueName: \"kubernetes.io/projected/90100a76-0fef-4430-b94f-05171ad876b1-kube-api-access-bbcxw\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:51.967147 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:51.967098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:52.253255 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:52.253136    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"]
Sep 17 18:50:52.255561 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:50:52.255529    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90100a76_0fef_4430_b94f_05171ad876b1.slice/crio-21db26e84cb961cc0632888e494e4035c040664650e06ecc03911f579152215f WatchSource:0}: Error finding container 21db26e84cb961cc0632888e494e4035c040664650e06ecc03911f579152215f: Status 404 returned error can't find the container with id 21db26e84cb961cc0632888e494e4035c040664650e06ecc03911f579152215f
Sep 17 18:50:52.657678 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:52.657586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" event={"ID":"90100a76-0fef-4430-b94f-05171ad876b1","Type":"ContainerStarted","Data":"32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3"}
Sep 17 18:50:52.658067 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:52.657674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" event={"ID":"90100a76-0fef-4430-b94f-05171ad876b1","Type":"ContainerStarted","Data":"21db26e84cb961cc0632888e494e4035c040664650e06ecc03911f579152215f"}
Sep 17 18:50:57.681843 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:57.681807    2577 generic.go:356] "Generic (PLEG): container finished" podID="90100a76-0fef-4430-b94f-05171ad876b1" containerID="32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3" exitCode=0
Sep 17 18:50:57.681843 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:57.681847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" event={"ID":"90100a76-0fef-4430-b94f-05171ad876b1","Type":"ContainerDied","Data":"32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3"}
Sep 17 18:50:58.695798 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:58.695760    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" event={"ID":"90100a76-0fef-4430-b94f-05171ad876b1","Type":"ContainerStarted","Data":"4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c"}
Sep 17 18:50:58.696166 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:58.695848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" event={"ID":"90100a76-0fef-4430-b94f-05171ad876b1","Type":"ContainerStarted","Data":"3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1"}
Sep 17 18:50:58.696166 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:58.696102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:50:58.722628 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:58.722574    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" podStartSLOduration=2.958302865 podStartE2EDuration="7.722559455s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.76425659s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:58.72182812 +0000 UTC m=+1293.768231050" watchObservedRunningTime="2026-09-17 18:50:58.722559455 +0000 UTC m=+1293.768962356"
Sep 17 18:50:59.700123 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:50:59.700093    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:51:05.711757 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:51:05.711719    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:52:36.267393 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:36.267362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:52:47.275097 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:47.275051    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.40:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:52:47.746572 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:47.746540    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"]
Sep 17 18:52:47.746801 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:47.746774    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="kserve-container" containerID="cri-o://3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1" gracePeriod=30
Sep 17 18:52:47.746883 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:47.746820    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="kube-rbac-proxy" containerID="cri-o://4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c" gracePeriod=30
Sep 17 18:52:48.211629 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:48.211590    2577 generic.go:356] "Generic (PLEG): container finished" podID="90100a76-0fef-4430-b94f-05171ad876b1" containerID="4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c" exitCode=2
Sep 17 18:52:48.211809 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:48.211658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" event={"ID":"90100a76-0fef-4430-b94f-05171ad876b1","Type":"ContainerDied","Data":"4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c"}
Sep 17 18:52:49.199390 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.199365    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:52:49.217801 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.217772    2577 generic.go:356] "Generic (PLEG): container finished" podID="90100a76-0fef-4430-b94f-05171ad876b1" containerID="3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1" exitCode=0
Sep 17 18:52:49.217947 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.217816    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"
Sep 17 18:52:49.217947 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.217851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" event={"ID":"90100a76-0fef-4430-b94f-05171ad876b1","Type":"ContainerDied","Data":"3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1"}
Sep 17 18:52:49.218080 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.217945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m" event={"ID":"90100a76-0fef-4430-b94f-05171ad876b1","Type":"ContainerDied","Data":"21db26e84cb961cc0632888e494e4035c040664650e06ecc03911f579152215f"}
Sep 17 18:52:49.218080 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.217986    2577 scope.go:117] "RemoveContainer" containerID="4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c"
Sep 17 18:52:49.227506 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.227490    2577 scope.go:117] "RemoveContainer" containerID="3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1"
Sep 17 18:52:49.236082 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.236056    2577 scope.go:117] "RemoveContainer" containerID="32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3"
Sep 17 18:52:49.243882 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.243865    2577 scope.go:117] "RemoveContainer" containerID="4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c"
Sep 17 18:52:49.244195 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:52:49.244176    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c\": container with ID starting with 4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c not found: ID does not exist" containerID="4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c"
Sep 17 18:52:49.244264 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.244206    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c"} err="failed to get container status \"4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c\": rpc error: code = NotFound desc = could not find container \"4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c\": container with ID starting with 4f9d8bb1c86efd9929061815ecc153b5fe8262cb0ebbd6e7aa47e29377954b2c not found: ID does not exist"
Sep 17 18:52:49.244264 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.244223    2577 scope.go:117] "RemoveContainer" containerID="3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1"
Sep 17 18:52:49.244474 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:52:49.244456    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1\": container with ID starting with 3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1 not found: ID does not exist" containerID="3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1"
Sep 17 18:52:49.244543 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.244484    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1"} err="failed to get container status \"3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1\": rpc error: code = NotFound desc = could not find container \"3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1\": container with ID starting with 3b93de02be80d5cb7a1ad9be762fc44439a976d4f04d278f4c564339eb2591a1 not found: ID does not exist"
Sep 17 18:52:49.244543 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.244510    2577 scope.go:117] "RemoveContainer" containerID="32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3"
Sep 17 18:52:49.244747 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:52:49.244730    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3\": container with ID starting with 32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3 not found: ID does not exist" containerID="32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3"
Sep 17 18:52:49.244788 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.244751    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3"} err="failed to get container status \"32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3\": rpc error: code = NotFound desc = could not find container \"32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3\": container with ID starting with 32e0fb06a6d3dceeb2970e948355f28d249ee587f4fb8e5cc3abe8297aea34a3 not found: ID does not exist"
Sep 17 18:52:49.324750 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.324659    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-kube-rbac-proxy-sar-config\") pod \"90100a76-0fef-4430-b94f-05171ad876b1\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") "
Sep 17 18:52:49.324750 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.324750    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbcxw\" (UniqueName: \"kubernetes.io/projected/90100a76-0fef-4430-b94f-05171ad876b1-kube-api-access-bbcxw\") pod \"90100a76-0fef-4430-b94f-05171ad876b1\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") "
Sep 17 18:52:49.325005 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.324785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90100a76-0fef-4430-b94f-05171ad876b1-kserve-provision-location\") pod \"90100a76-0fef-4430-b94f-05171ad876b1\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") "
Sep 17 18:52:49.325005 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.324809    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-cabundle-cert\") pod \"90100a76-0fef-4430-b94f-05171ad876b1\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") "
Sep 17 18:52:49.325005 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.324833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90100a76-0fef-4430-b94f-05171ad876b1-proxy-tls\") pod \"90100a76-0fef-4430-b94f-05171ad876b1\" (UID: \"90100a76-0fef-4430-b94f-05171ad876b1\") "
Sep 17 18:52:49.325206 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.325180    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "90100a76-0fef-4430-b94f-05171ad876b1" (UID: "90100a76-0fef-4430-b94f-05171ad876b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:49.325256 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.325197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "90100a76-0fef-4430-b94f-05171ad876b1" (UID: "90100a76-0fef-4430-b94f-05171ad876b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:49.327035 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.327012    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90100a76-0fef-4430-b94f-05171ad876b1-kube-api-access-bbcxw" (OuterVolumeSpecName: "kube-api-access-bbcxw") pod "90100a76-0fef-4430-b94f-05171ad876b1" (UID: "90100a76-0fef-4430-b94f-05171ad876b1"). InnerVolumeSpecName "kube-api-access-bbcxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:52:49.327113 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.327064    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90100a76-0fef-4430-b94f-05171ad876b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90100a76-0fef-4430-b94f-05171ad876b1" (UID: "90100a76-0fef-4430-b94f-05171ad876b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:52:49.329547 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.329524    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90100a76-0fef-4430-b94f-05171ad876b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90100a76-0fef-4430-b94f-05171ad876b1" (UID: "90100a76-0fef-4430-b94f-05171ad876b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:49.425694 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.425660    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:52:49.425694 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.425690    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90100a76-0fef-4430-b94f-05171ad876b1-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:52:49.425694 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.425700    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90100a76-0fef-4430-b94f-05171ad876b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:52:49.425959 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.425711    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbcxw\" (UniqueName: \"kubernetes.io/projected/90100a76-0fef-4430-b94f-05171ad876b1-kube-api-access-bbcxw\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:52:49.425959 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.425721    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90100a76-0fef-4430-b94f-05171ad876b1-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:52:49.541210 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.541181    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"]
Sep 17 18:52:49.545204 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:49.545181    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gb86m"]
Sep 17 18:52:51.237879 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:51.237845    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90100a76-0fef-4430-b94f-05171ad876b1" path="/var/lib/kubelet/pods/90100a76-0fef-4430-b94f-05171ad876b1/volumes"
Sep 17 18:52:52.956512 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.956477    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"]
Sep 17 18:52:52.956973 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.956878    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="storage-initializer"
Sep 17 18:52:52.956973 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.956891    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="storage-initializer"
Sep 17 18:52:52.956973 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.956914    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="kserve-container"
Sep 17 18:52:52.956973 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.956922    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="kserve-container"
Sep 17 18:52:52.956973 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.956929    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="kube-rbac-proxy"
Sep 17 18:52:52.956973 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.956935    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="kube-rbac-proxy"
Sep 17 18:52:52.957173 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.957008    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="kube-rbac-proxy"
Sep 17 18:52:52.957173 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.957016    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="90100a76-0fef-4430-b94f-05171ad876b1" containerName="kserve-container"
Sep 17 18:52:52.961919 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.961888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:52.964574 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.964550    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 18:52:52.964723 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.964610    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:52:52.964723 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.964651    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:52:52.964723 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.964675    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:52:52.964723 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.964689    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 18:52:52.964723 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.964703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 18:52:52.964998 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.964612    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:52:52.970708 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:52.970682    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"]
Sep 17 18:52:53.058759 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.058720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.058961 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.058767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f98kl\" (UniqueName: \"kubernetes.io/projected/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-api-access-f98kl\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.058961 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.058866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.058961 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.058898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.059097 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.058976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.160078 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.160040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f98kl\" (UniqueName: \"kubernetes.io/projected/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-api-access-f98kl\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.160292 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.160113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.160292 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.160139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.160292 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.160172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.160292 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.160240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.160664 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.160638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.160838 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.160817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.160935 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.160859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.162608 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.162589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.168461 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.168440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f98kl\" (UniqueName: \"kubernetes.io/projected/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-api-access-f98kl\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.273866 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.273771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:52:53.549408 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:53.549342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"]
Sep 17 18:52:53.552027 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:52:53.551999    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2dd8fc_24ff_419c_9afa_4f911e07fd99.slice/crio-836c250ba455f88052ff837cf0960a8bebedddc76a0a3302f54b55695be72998 WatchSource:0}: Error finding container 836c250ba455f88052ff837cf0960a8bebedddc76a0a3302f54b55695be72998: Status 404 returned error can't find the container with id 836c250ba455f88052ff837cf0960a8bebedddc76a0a3302f54b55695be72998
Sep 17 18:52:54.242415 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:54.242373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" event={"ID":"0a2dd8fc-24ff-419c-9afa-4f911e07fd99","Type":"ContainerStarted","Data":"56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798"}
Sep 17 18:52:54.242883 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:52:54.242439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" event={"ID":"0a2dd8fc-24ff-419c-9afa-4f911e07fd99","Type":"ContainerStarted","Data":"836c250ba455f88052ff837cf0960a8bebedddc76a0a3302f54b55695be72998"}
Sep 17 18:53:03.283460 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:03.283422    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerID="56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798" exitCode=0
Sep 17 18:53:03.283833 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:03.283494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" event={"ID":"0a2dd8fc-24ff-419c-9afa-4f911e07fd99","Type":"ContainerDied","Data":"56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798"}
Sep 17 18:53:04.290898 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:04.290863    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" event={"ID":"0a2dd8fc-24ff-419c-9afa-4f911e07fd99","Type":"ContainerStarted","Data":"a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417"}
Sep 17 18:53:04.291341 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:04.290945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" event={"ID":"0a2dd8fc-24ff-419c-9afa-4f911e07fd99","Type":"ContainerStarted","Data":"ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a"}
Sep 17 18:53:04.291341 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:04.291139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:53:04.311755 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:04.311707    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" podStartSLOduration=2.752564066 podStartE2EDuration="12.3116924s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.559128334s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:53:04.310183862 +0000 UTC m=+1419.356586793" watchObservedRunningTime="2026-09-17 18:53:04.3116924 +0000 UTC m=+1419.358095302"
Sep 17 18:53:05.295469 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:05.295434    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:53:05.297054 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:05.297020    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:53:06.301108 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:06.301070    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:53:11.306546 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:11.306465    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:53:11.307198 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:11.306968    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:53:21.307948 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:21.307921    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:53:35.438165 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:35.438127    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"]
Sep 17 18:53:35.438887 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:35.438415    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kserve-container" containerID="cri-o://ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a" gracePeriod=30
Sep 17 18:53:35.438887 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:35.438452    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kube-rbac-proxy" containerID="cri-o://a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417" gracePeriod=30
Sep 17 18:53:36.302064 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:36.302020    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 17 18:53:36.459837 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:36.459808    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerID="a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417" exitCode=2
Sep 17 18:53:36.460216 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:36.459883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" event={"ID":"0a2dd8fc-24ff-419c-9afa-4f911e07fd99","Type":"ContainerDied","Data":"a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417"}
Sep 17 18:53:37.083415 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.083392    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:53:37.257135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.257110    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f98kl\" (UniqueName: \"kubernetes.io/projected/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-api-access-f98kl\") pod \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") "
Sep 17 18:53:37.257321 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.257201    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kserve-provision-location\") pod \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") "
Sep 17 18:53:37.257321 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.257243    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-rbac-proxy-sar-config\") pod \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") "
Sep 17 18:53:37.257321 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.257278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-cabundle-cert\") pod \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") "
Sep 17 18:53:37.257321 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.257303    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-proxy-tls\") pod \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\" (UID: \"0a2dd8fc-24ff-419c-9afa-4f911e07fd99\") "
Sep 17 18:53:37.257687 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.257661    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a2dd8fc-24ff-419c-9afa-4f911e07fd99" (UID: "0a2dd8fc-24ff-419c-9afa-4f911e07fd99"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:37.257801 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.257659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0a2dd8fc-24ff-419c-9afa-4f911e07fd99" (UID: "0a2dd8fc-24ff-419c-9afa-4f911e07fd99"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:37.258135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.258116    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a2dd8fc-24ff-419c-9afa-4f911e07fd99" (UID: "0a2dd8fc-24ff-419c-9afa-4f911e07fd99"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:37.259291 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.259270    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a2dd8fc-24ff-419c-9afa-4f911e07fd99" (UID: "0a2dd8fc-24ff-419c-9afa-4f911e07fd99"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:37.259357 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.259275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-api-access-f98kl" (OuterVolumeSpecName: "kube-api-access-f98kl") pod "0a2dd8fc-24ff-419c-9afa-4f911e07fd99" (UID: "0a2dd8fc-24ff-419c-9afa-4f911e07fd99"). InnerVolumeSpecName "kube-api-access-f98kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:37.358302 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.358274    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:53:37.358302 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.358299    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:53:37.358450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.358309    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:53:37.358450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.358318    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:53:37.358450 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.358326    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f98kl\" (UniqueName: \"kubernetes.io/projected/0a2dd8fc-24ff-419c-9afa-4f911e07fd99-kube-api-access-f98kl\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:53:37.465775 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.465741    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerID="ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a" exitCode=0
Sep 17 18:53:37.466135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.465805    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"
Sep 17 18:53:37.466135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.465820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" event={"ID":"0a2dd8fc-24ff-419c-9afa-4f911e07fd99","Type":"ContainerDied","Data":"ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a"}
Sep 17 18:53:37.466135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.465878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4" event={"ID":"0a2dd8fc-24ff-419c-9afa-4f911e07fd99","Type":"ContainerDied","Data":"836c250ba455f88052ff837cf0960a8bebedddc76a0a3302f54b55695be72998"}
Sep 17 18:53:37.466135 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.465919    2577 scope.go:117] "RemoveContainer" containerID="a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417"
Sep 17 18:53:37.480723 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.480703    2577 scope.go:117] "RemoveContainer" containerID="ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a"
Sep 17 18:53:37.488047 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.488033    2577 scope.go:117] "RemoveContainer" containerID="56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798"
Sep 17 18:53:37.492652 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.492630    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"]
Sep 17 18:53:37.496056 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.496025    2577 scope.go:117] "RemoveContainer" containerID="a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417"
Sep 17 18:53:37.496316 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:53:37.496292    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417\": container with ID starting with a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417 not found: ID does not exist" containerID="a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417"
Sep 17 18:53:37.496392 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.496328    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417"} err="failed to get container status \"a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417\": rpc error: code = NotFound desc = could not find container \"a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417\": container with ID starting with a9ddefe166f3dae063d88f973fa8a88e5045a06066924f51ae70d8c06c642417 not found: ID does not exist"
Sep 17 18:53:37.496392 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.496351    2577 scope.go:117] "RemoveContainer" containerID="ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a"
Sep 17 18:53:37.496504 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.496488    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z7kw4"]
Sep 17 18:53:37.496647 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:53:37.496625    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a\": container with ID starting with ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a not found: ID does not exist" containerID="ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a"
Sep 17 18:53:37.496689 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.496657    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a"} err="failed to get container status \"ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a\": rpc error: code = NotFound desc = could not find container \"ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a\": container with ID starting with ad4f4cbfcc4d5cc74045110b3ca0a35004672503d313de074349483673cf950a not found: ID does not exist"
Sep 17 18:53:37.496689 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.496680    2577 scope.go:117] "RemoveContainer" containerID="56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798"
Sep 17 18:53:37.497172 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:53:37.497149    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798\": container with ID starting with 56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798 not found: ID does not exist" containerID="56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798"
Sep 17 18:53:37.497252 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:37.497176    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798"} err="failed to get container status \"56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798\": rpc error: code = NotFound desc = could not find container \"56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798\": container with ID starting with 56305efc6bb1ca7472a0db771508ee3da243c23a6884fb338b9d15499c440798 not found: ID does not exist"
Sep 17 18:53:39.238492 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:39.238457    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" path="/var/lib/kubelet/pods/0a2dd8fc-24ff-419c-9afa-4f911e07fd99/volumes"
Sep 17 18:53:40.654921 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.654868    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"]
Sep 17 18:53:40.655285 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.655236    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kube-rbac-proxy"
Sep 17 18:53:40.655285 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.655248    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kube-rbac-proxy"
Sep 17 18:53:40.655285 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.655264    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kserve-container"
Sep 17 18:53:40.655285 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.655270    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kserve-container"
Sep 17 18:53:40.655285 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.655279    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="storage-initializer"
Sep 17 18:53:40.655285 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.655285    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="storage-initializer"
Sep 17 18:53:40.655515 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.655341    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kserve-container"
Sep 17 18:53:40.655515 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.655353    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2dd8fc-24ff-419c-9afa-4f911e07fd99" containerName="kube-rbac-proxy"
Sep 17 18:53:40.660586 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.660563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.663051 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.663032    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 18:53:40.663747 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.663732    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:53:40.663861 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.663843    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:53:40.663931 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.663868    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 18:53:40.664100 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.664085    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:53:40.664140 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.664108    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 18:53:40.664339 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.664326    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:53:40.671012 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.670991    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"]
Sep 17 18:53:40.786647 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.786608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.786841 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.786660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.786841 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.786728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcn9w\" (UniqueName: \"kubernetes.io/projected/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-api-access-xcn9w\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.786841 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.786785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.786841 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.786837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.887327 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.887289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.887521 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.887339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.887521 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.887369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcn9w\" (UniqueName: \"kubernetes.io/projected/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-api-access-xcn9w\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.887521 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.887388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.887521 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.887424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.887885 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.887859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.888114 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.888093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.888182 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.888111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.889969 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.889941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.900064 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.900038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcn9w\" (UniqueName: \"kubernetes.io/projected/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-api-access-xcn9w\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:40.970980 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:40.970956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:41.249048 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:41.249014    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"]
Sep 17 18:53:41.249732 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:53:41.249707    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4faa45_7903_463c_a5e8_c3d9ddff76cd.slice/crio-4185d13f8b852df0b6682cdacc301c29278c48e662256d973276429e84d32515 WatchSource:0}: Error finding container 4185d13f8b852df0b6682cdacc301c29278c48e662256d973276429e84d32515: Status 404 returned error can't find the container with id 4185d13f8b852df0b6682cdacc301c29278c48e662256d973276429e84d32515
Sep 17 18:53:41.485886 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:41.485853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" event={"ID":"0b4faa45-7903-463c-a5e8-c3d9ddff76cd","Type":"ContainerStarted","Data":"ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177"}
Sep 17 18:53:41.486067 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:41.485926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" event={"ID":"0b4faa45-7903-463c-a5e8-c3d9ddff76cd","Type":"ContainerStarted","Data":"4185d13f8b852df0b6682cdacc301c29278c48e662256d973276429e84d32515"}
Sep 17 18:53:46.509768 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:46.509730    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerID="ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177" exitCode=0
Sep 17 18:53:46.510218 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:46.509802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" event={"ID":"0b4faa45-7903-463c-a5e8-c3d9ddff76cd","Type":"ContainerDied","Data":"ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177"}
Sep 17 18:53:47.517124 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:47.517088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" event={"ID":"0b4faa45-7903-463c-a5e8-c3d9ddff76cd","Type":"ContainerStarted","Data":"ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551"}
Sep 17 18:53:47.517604 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:47.517160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" event={"ID":"0b4faa45-7903-463c-a5e8-c3d9ddff76cd","Type":"ContainerStarted","Data":"ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b"}
Sep 17 18:53:47.517604 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:47.517369    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:47.537382 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:47.537327    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" podStartSLOduration=3.12262124 podStartE2EDuration="7.537307465s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.414686225s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:53:47.535751895 +0000 UTC m=+1462.582154836" watchObservedRunningTime="2026-09-17 18:53:47.537307465 +0000 UTC m=+1462.583710371"
Sep 17 18:53:48.522459 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:48.522420    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:48.523815 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:48.523785    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:53:49.533650 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:49.533610    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:53:54.535354 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:54.535327    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:53:54.535919 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:53:54.535876    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:54:04.536756 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:04.536728    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:54:16.025222 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:16.025182    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"]
Sep 17 18:54:16.025706 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:16.025506    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kserve-container" containerID="cri-o://ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b" gracePeriod=30
Sep 17 18:54:16.025706 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:16.025545    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kube-rbac-proxy" containerID="cri-o://ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551" gracePeriod=30
Sep 17 18:54:16.669746 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:16.669714    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerID="ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551" exitCode=2
Sep 17 18:54:16.669936 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:16.669786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" event={"ID":"0b4faa45-7903-463c-a5e8-c3d9ddff76cd","Type":"ContainerDied","Data":"ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551"}
Sep 17 18:54:17.477672 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.477648    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:54:17.604277 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.604190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kserve-provision-location\") pod \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") "
Sep 17 18:54:17.604277 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.604247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcn9w\" (UniqueName: \"kubernetes.io/projected/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-api-access-xcn9w\") pod \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") "
Sep 17 18:54:17.604502 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.604313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-cabundle-cert\") pod \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") "
Sep 17 18:54:17.604502 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.604379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-proxy-tls\") pod \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") "
Sep 17 18:54:17.604502 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.604411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-rbac-proxy-sar-config\") pod \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\" (UID: \"0b4faa45-7903-463c-a5e8-c3d9ddff76cd\") "
Sep 17 18:54:17.604727 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.604701    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b4faa45-7903-463c-a5e8-c3d9ddff76cd" (UID: "0b4faa45-7903-463c-a5e8-c3d9ddff76cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:17.604826 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.604768    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b4faa45-7903-463c-a5e8-c3d9ddff76cd" (UID: "0b4faa45-7903-463c-a5e8-c3d9ddff76cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:17.605249 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.605229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b4faa45-7903-463c-a5e8-c3d9ddff76cd" (UID: "0b4faa45-7903-463c-a5e8-c3d9ddff76cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:17.606466 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.606441    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b4faa45-7903-463c-a5e8-c3d9ddff76cd" (UID: "0b4faa45-7903-463c-a5e8-c3d9ddff76cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:17.606554 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.606513    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-api-access-xcn9w" (OuterVolumeSpecName: "kube-api-access-xcn9w") pod "0b4faa45-7903-463c-a5e8-c3d9ddff76cd" (UID: "0b4faa45-7903-463c-a5e8-c3d9ddff76cd"). InnerVolumeSpecName "kube-api-access-xcn9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:17.676435 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.676403    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerID="ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b" exitCode=0
Sep 17 18:54:17.676569 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.676469    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"
Sep 17 18:54:17.676569 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.676483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" event={"ID":"0b4faa45-7903-463c-a5e8-c3d9ddff76cd","Type":"ContainerDied","Data":"ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b"}
Sep 17 18:54:17.676569 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.676541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc" event={"ID":"0b4faa45-7903-463c-a5e8-c3d9ddff76cd","Type":"ContainerDied","Data":"4185d13f8b852df0b6682cdacc301c29278c48e662256d973276429e84d32515"}
Sep 17 18:54:17.676687 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.676573    2577 scope.go:117] "RemoveContainer" containerID="ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551"
Sep 17 18:54:17.685390 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.685371    2577 scope.go:117] "RemoveContainer" containerID="ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b"
Sep 17 18:54:17.693125 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.693109    2577 scope.go:117] "RemoveContainer" containerID="ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177"
Sep 17 18:54:17.699915 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.699880    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"]
Sep 17 18:54:17.700643 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.700630    2577 scope.go:117] "RemoveContainer" containerID="ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551"
Sep 17 18:54:17.700879 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:54:17.700862    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551\": container with ID starting with ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551 not found: ID does not exist" containerID="ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551"
Sep 17 18:54:17.700943 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.700887    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551"} err="failed to get container status \"ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551\": rpc error: code = NotFound desc = could not find container \"ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551\": container with ID starting with ecd583dec6e88d7d37b0c0079f277ffcc26b6ddcdc6449f695b6be85d7634551 not found: ID does not exist"
Sep 17 18:54:17.700943 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.700914    2577 scope.go:117] "RemoveContainer" containerID="ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b"
Sep 17 18:54:17.701132 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:54:17.701116    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b\": container with ID starting with ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b not found: ID does not exist" containerID="ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b"
Sep 17 18:54:17.701191 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.701138    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b"} err="failed to get container status \"ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b\": rpc error: code = NotFound desc = could not find container \"ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b\": container with ID starting with ab0b9e052d29b6ea9669fdf3d247c6f20b91a76850540d34c09038b1eedb871b not found: ID does not exist"
Sep 17 18:54:17.701191 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.701151    2577 scope.go:117] "RemoveContainer" containerID="ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177"
Sep 17 18:54:17.701418 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:54:17.701400    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177\": container with ID starting with ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177 not found: ID does not exist" containerID="ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177"
Sep 17 18:54:17.701457 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.701425    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177"} err="failed to get container status \"ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177\": rpc error: code = NotFound desc = could not find container \"ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177\": container with ID starting with ebec7a64392aa9b439d5ae53b4d2e754a74be423af37ac443a1f57eef55d6177 not found: ID does not exist"
Sep 17 18:54:17.704001 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.703978    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-m9sjc"]
Sep 17 18:54:17.705570 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.705555    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcn9w\" (UniqueName: \"kubernetes.io/projected/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-api-access-xcn9w\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:54:17.705634 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.705572    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:54:17.705634 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.705581    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:54:17.705634 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.705589    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:54:17.705634 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:17.705597    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b4faa45-7903-463c-a5e8-c3d9ddff76cd-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:54:19.238369 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:19.238336    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" path="/var/lib/kubelet/pods/0b4faa45-7903-463c-a5e8-c3d9ddff76cd/volumes"
Sep 17 18:54:21.434385 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.434349    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"]
Sep 17 18:54:21.434736 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.434706    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kserve-container"
Sep 17 18:54:21.434736 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.434717    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kserve-container"
Sep 17 18:54:21.434736 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.434730    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="storage-initializer"
Sep 17 18:54:21.434736 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.434736    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="storage-initializer"
Sep 17 18:54:21.434861 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.434746    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kube-rbac-proxy"
Sep 17 18:54:21.434861 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.434752    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kube-rbac-proxy"
Sep 17 18:54:21.434861 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.434811    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kserve-container"
Sep 17 18:54:21.434861 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.434818    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4faa45-7903-463c-a5e8-c3d9ddff76cd" containerName="kube-rbac-proxy"
Sep 17 18:54:21.439424 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.439407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.446736 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.446719    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:54:21.447617 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.447602    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:54:21.447789 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.447773    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:54:21.447850 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.447835    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:54:21.447895 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.447854    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 18:54:21.447895 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.447861    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:54:21.447993 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.447970    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:54:21.464333 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.464308    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"]
Sep 17 18:54:21.538150 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.538122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.538271 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.538168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18804994-2add-4d6e-981f-9f0f78cd0d9b-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.538271 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.538233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18804994-2add-4d6e-981f-9f0f78cd0d9b-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.538345 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.538280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqgz\" (UniqueName: \"kubernetes.io/projected/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-api-access-sgqgz\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.538345 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.538300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.639428 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.639401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqgz\" (UniqueName: \"kubernetes.io/projected/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-api-access-sgqgz\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.639529 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.639446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.639529 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.639496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.639604 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.639540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18804994-2add-4d6e-981f-9f0f78cd0d9b-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.639604 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.639580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18804994-2add-4d6e-981f-9f0f78cd0d9b-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.639958 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.639942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18804994-2add-4d6e-981f-9f0f78cd0d9b-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.640284 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.640263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.640323 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.640271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.641982 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.641961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18804994-2add-4d6e-981f-9f0f78cd0d9b-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.648355 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.648328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqgz\" (UniqueName: \"kubernetes.io/projected/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-api-access-sgqgz\") pod \"isvc-canary-create-predictor-66d699644d-8dgsk\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:21.749943 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:21.749872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:22.026597 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:22.026571    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"]
Sep 17 18:54:22.028354 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:54:22.028325    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18804994_2add_4d6e_981f_9f0f78cd0d9b.slice/crio-bf9566aebfcc77c213270a50538ad43e138cb04c10282f7b8d6cd20a6ae576af WatchSource:0}: Error finding container bf9566aebfcc77c213270a50538ad43e138cb04c10282f7b8d6cd20a6ae576af: Status 404 returned error can't find the container with id bf9566aebfcc77c213270a50538ad43e138cb04c10282f7b8d6cd20a6ae576af
Sep 17 18:54:22.704094 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:22.704057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" event={"ID":"18804994-2add-4d6e-981f-9f0f78cd0d9b","Type":"ContainerStarted","Data":"3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb"}
Sep 17 18:54:22.704463 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:22.704132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" event={"ID":"18804994-2add-4d6e-981f-9f0f78cd0d9b","Type":"ContainerStarted","Data":"bf9566aebfcc77c213270a50538ad43e138cb04c10282f7b8d6cd20a6ae576af"}
Sep 17 18:54:25.721073 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:25.721041    2577 generic.go:356] "Generic (PLEG): container finished" podID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerID="3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb" exitCode=0
Sep 17 18:54:25.721384 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:25.721117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" event={"ID":"18804994-2add-4d6e-981f-9f0f78cd0d9b","Type":"ContainerDied","Data":"3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb"}
Sep 17 18:54:25.722293 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:25.722272    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:54:39.810963 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:39.810928    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" event={"ID":"18804994-2add-4d6e-981f-9f0f78cd0d9b","Type":"ContainerStarted","Data":"02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f"}
Sep 17 18:54:39.811399 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:39.810994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" event={"ID":"18804994-2add-4d6e-981f-9f0f78cd0d9b","Type":"ContainerStarted","Data":"452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6"}
Sep 17 18:54:39.811399 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:39.811262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:39.811527 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:39.811402    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:39.813023 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:39.812994    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:54:39.838389 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:39.838335    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podStartSLOduration=2.165696739 podStartE2EDuration="18.838315955s" totalImagesPullingTime="13.087652482s" totalInitContainerRuntime="3.584966734s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:39.837162065 +0000 UTC m=+1514.883564990" watchObservedRunningTime="2026-09-17 18:54:39.838315955 +0000 UTC m=+1514.884718857"
Sep 17 18:54:40.815631 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:40.815584    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:54:45.819989 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:45.819954    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:54:45.820526 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:45.820493    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:54:55.820613 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:54:55.820565    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:55:05.822533 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:05.822485    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:55:15.820497 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:15.820448    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:55:25.821134 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:25.821094    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:55:35.820468 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:35.820422    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:55:45.820848 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:45.820800    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:55:51.128716 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:51.128683    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"]
Sep 17 18:55:51.129233 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:51.128951    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" containerID="cri-o://452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6" gracePeriod=30
Sep 17 18:55:51.129233 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:51.128999    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kube-rbac-proxy" containerID="cri-o://02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f" gracePeriod=30
Sep 17 18:55:51.129233 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:51.129154    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:55:52.147560 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:52.147526    2577 generic.go:356] "Generic (PLEG): container finished" podID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerID="02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f" exitCode=2
Sep 17 18:55:52.147973 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:52.147606    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" event={"ID":"18804994-2add-4d6e-981f-9f0f78cd0d9b","Type":"ContainerDied","Data":"02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f"}
Sep 17 18:55:55.870366 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.870343    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:55:55.895873 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.895826    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18804994-2add-4d6e-981f-9f0f78cd0d9b-proxy-tls\") pod \"18804994-2add-4d6e-981f-9f0f78cd0d9b\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") "
Sep 17 18:55:55.895873 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.895862    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-rbac-proxy-sar-config\") pod \"18804994-2add-4d6e-981f-9f0f78cd0d9b\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") "
Sep 17 18:55:55.895996 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.895881    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgqgz\" (UniqueName: \"kubernetes.io/projected/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-api-access-sgqgz\") pod \"18804994-2add-4d6e-981f-9f0f78cd0d9b\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") "
Sep 17 18:55:55.895996 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.895933    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-cabundle-cert\") pod \"18804994-2add-4d6e-981f-9f0f78cd0d9b\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") "
Sep 17 18:55:55.895996 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.895960    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18804994-2add-4d6e-981f-9f0f78cd0d9b-kserve-provision-location\") pod \"18804994-2add-4d6e-981f-9f0f78cd0d9b\" (UID: \"18804994-2add-4d6e-981f-9f0f78cd0d9b\") "
Sep 17 18:55:55.896305 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.896268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18804994-2add-4d6e-981f-9f0f78cd0d9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18804994-2add-4d6e-981f-9f0f78cd0d9b" (UID: "18804994-2add-4d6e-981f-9f0f78cd0d9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:55:55.896305 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.896299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18804994-2add-4d6e-981f-9f0f78cd0d9b" (UID: "18804994-2add-4d6e-981f-9f0f78cd0d9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:55:55.896452 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.896301    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18804994-2add-4d6e-981f-9f0f78cd0d9b" (UID: "18804994-2add-4d6e-981f-9f0f78cd0d9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:55:55.897882 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.897857    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18804994-2add-4d6e-981f-9f0f78cd0d9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18804994-2add-4d6e-981f-9f0f78cd0d9b" (UID: "18804994-2add-4d6e-981f-9f0f78cd0d9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:55:55.898025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.897936    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-api-access-sgqgz" (OuterVolumeSpecName: "kube-api-access-sgqgz") pod "18804994-2add-4d6e-981f-9f0f78cd0d9b" (UID: "18804994-2add-4d6e-981f-9f0f78cd0d9b"). InnerVolumeSpecName "kube-api-access-sgqgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:55:55.997567 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.997530    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18804994-2add-4d6e-981f-9f0f78cd0d9b-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:55:55.997567 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.997561    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18804994-2add-4d6e-981f-9f0f78cd0d9b-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:55:55.997567 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.997570    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:55:55.997797 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.997579    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgqgz\" (UniqueName: \"kubernetes.io/projected/18804994-2add-4d6e-981f-9f0f78cd0d9b-kube-api-access-sgqgz\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:55:55.997797 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:55.997590    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18804994-2add-4d6e-981f-9f0f78cd0d9b-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:55:56.168260 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.168171    2577 generic.go:356] "Generic (PLEG): container finished" podID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerID="452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6" exitCode=139
Sep 17 18:55:56.168260 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.168215    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" event={"ID":"18804994-2add-4d6e-981f-9f0f78cd0d9b","Type":"ContainerDied","Data":"452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6"}
Sep 17 18:55:56.168260 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.168220    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"
Sep 17 18:55:56.168492 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.168271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" event={"ID":"18804994-2add-4d6e-981f-9f0f78cd0d9b","Type":"ContainerDied","Data":"bf9566aebfcc77c213270a50538ad43e138cb04c10282f7b8d6cd20a6ae576af"}
Sep 17 18:55:56.168492 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.168298    2577 scope.go:117] "RemoveContainer" containerID="02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f"
Sep 17 18:55:56.177409 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.177390    2577 scope.go:117] "RemoveContainer" containerID="452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6"
Sep 17 18:55:56.184684 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.184666    2577 scope.go:117] "RemoveContainer" containerID="3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb"
Sep 17 18:55:56.192299 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.192284    2577 scope.go:117] "RemoveContainer" containerID="02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f"
Sep 17 18:55:56.192591 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:55:56.192573    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f\": container with ID starting with 02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f not found: ID does not exist" containerID="02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f"
Sep 17 18:55:56.192656 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.192598    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f"} err="failed to get container status \"02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f\": rpc error: code = NotFound desc = could not find container \"02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f\": container with ID starting with 02d3a0f960995630a5f31f1585912897a80ca5c9e1f456f7ccd49a3dc5cc397f not found: ID does not exist"
Sep 17 18:55:56.192656 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.192614    2577 scope.go:117] "RemoveContainer" containerID="452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6"
Sep 17 18:55:56.192779 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.192760    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"]
Sep 17 18:55:56.192871 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:55:56.192856    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6\": container with ID starting with 452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6 not found: ID does not exist" containerID="452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6"
Sep 17 18:55:56.192925 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.192876    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6"} err="failed to get container status \"452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6\": rpc error: code = NotFound desc = could not find container \"452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6\": container with ID starting with 452d106c9dc0715fb2bd484bff17af3e4f5472aae320a132078fda43190cd1c6 not found: ID does not exist"
Sep 17 18:55:56.192925 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.192891    2577 scope.go:117] "RemoveContainer" containerID="3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb"
Sep 17 18:55:56.193155 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:55:56.193135    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb\": container with ID starting with 3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb not found: ID does not exist" containerID="3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb"
Sep 17 18:55:56.193214 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.193160    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb"} err="failed to get container status \"3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb\": rpc error: code = NotFound desc = could not find container \"3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb\": container with ID starting with 3ce9915ac12eece836b1e006bf268c79c6523e51ca319661242c68f38e875dcb not found: ID does not exist"
Sep 17 18:55:56.197594 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.197574    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk"]
Sep 17 18:55:56.816555 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:56.816503    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-8dgsk" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": context deadline exceeded"
Sep 17 18:55:57.239088 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:55:57.239056    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" path="/var/lib/kubelet/pods/18804994-2add-4d6e-981f-9f0f78cd0d9b/volumes"
Sep 17 18:56:06.529069 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.529033    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"]
Sep 17 18:56:06.529427 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.529386    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="storage-initializer"
Sep 17 18:56:06.529427 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.529397    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="storage-initializer"
Sep 17 18:56:06.529427 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.529408    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container"
Sep 17 18:56:06.529427 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.529414    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container"
Sep 17 18:56:06.529575 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.529431    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kube-rbac-proxy"
Sep 17 18:56:06.529575 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.529437    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kube-rbac-proxy"
Sep 17 18:56:06.529575 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.529491    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kube-rbac-proxy"
Sep 17 18:56:06.529575 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.529503    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18804994-2add-4d6e-981f-9f0f78cd0d9b" containerName="kserve-container"
Sep 17 18:56:06.532537 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.532521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.535148 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.535126    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:56:06.535281 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.535241    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 18:56:06.535347 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.535316    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:56:06.535570 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.535554    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:56:06.535669 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.535558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:56:06.536122 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.536104    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:56:06.537449 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.537432    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:56:06.556136 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.556114    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"]
Sep 17 18:56:06.584227 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.584201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.584343 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.584261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.584343 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.584294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4s86\" (UniqueName: \"kubernetes.io/projected/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-api-access-k4s86\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.584416 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.584360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.584451 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.584417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4773d-8793-4bd4-8f15-b074bf4a18cc-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.677685 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.677656    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"]
Sep 17 18:56:06.680287 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.680271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.685413 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.685386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.685500 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.685426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4s86\" (UniqueName: \"kubernetes.io/projected/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-api-access-k4s86\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.685500 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.685446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.685608 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.685497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4773d-8793-4bd4-8f15-b074bf4a18cc-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.685608 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.685538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.685701 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:56:06.685668    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 18:56:06.685745 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:56:06.685732    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45b4773d-8793-4bd4-8f15-b074bf4a18cc-proxy-tls podName:45b4773d-8793-4bd4-8f15-b074bf4a18cc nodeName:}" failed. No retries permitted until 2026-09-17 18:56:07.185713446 +0000 UTC m=+1602.232116341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45b4773d-8793-4bd4-8f15-b074bf4a18cc-proxy-tls") pod "isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" (UID: "45b4773d-8793-4bd4-8f15-b074bf4a18cc") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 18:56:06.685919 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.685888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.686201 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.686175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.686302 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.686176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.687547 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.687528    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:56:06.699305 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.699281    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"]
Sep 17 18:56:06.705833 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.705815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4s86\" (UniqueName: \"kubernetes.io/projected/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-api-access-k4s86\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:06.786983 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.786898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.786983 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.786965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467w5\" (UniqueName: \"kubernetes.io/projected/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-api-access-467w5\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.787121 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.787019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.787121 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.787042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221174c0-7107-451a-b26b-c09ba0f5fb74-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.787121 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.787076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221174c0-7107-451a-b26b-c09ba0f5fb74-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.887870 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.887843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.888000 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.887874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221174c0-7107-451a-b26b-c09ba0f5fb74-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.888000 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.887899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221174c0-7107-451a-b26b-c09ba0f5fb74-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.888090 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.888006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.888090 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.888046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-467w5\" (UniqueName: \"kubernetes.io/projected/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-api-access-467w5\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.888409 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.888389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221174c0-7107-451a-b26b-c09ba0f5fb74-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.888674 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.888654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.888722 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.888660    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.890452 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.890432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221174c0-7107-451a-b26b-c09ba0f5fb74-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.906209 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.906186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-467w5\" (UniqueName: \"kubernetes.io/projected/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-api-access-467w5\") pod \"isvc-canary-promote-predictor-dfdfd784d-ml2zr\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:06.989715 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:06.989678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:07.191010 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:07.190975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4773d-8793-4bd4-8f15-b074bf4a18cc-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:07.193264 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:07.193236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4773d-8793-4bd4-8f15-b074bf4a18cc-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:07.264889 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:07.264864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"]
Sep 17 18:56:07.267246 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:56:07.267219    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221174c0_7107_451a_b26b_c09ba0f5fb74.slice/crio-6999f6317908528a2573b225797c92ab0336e7fbbb09441d1612752dfa336ac4 WatchSource:0}: Error finding container 6999f6317908528a2573b225797c92ab0336e7fbbb09441d1612752dfa336ac4: Status 404 returned error can't find the container with id 6999f6317908528a2573b225797c92ab0336e7fbbb09441d1612752dfa336ac4
Sep 17 18:56:07.443270 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:07.443231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:07.714623 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:07.714405    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"]
Sep 17 18:56:07.718117 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:56:07.718081    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4773d_8793_4bd4_8f15_b074bf4a18cc.slice/crio-ab188f81c117115bd9559d51768642c7b38045e7f6decd7eeeb687e7d3fa5b7c WatchSource:0}: Error finding container ab188f81c117115bd9559d51768642c7b38045e7f6decd7eeeb687e7d3fa5b7c: Status 404 returned error can't find the container with id ab188f81c117115bd9559d51768642c7b38045e7f6decd7eeeb687e7d3fa5b7c
Sep 17 18:56:08.227102 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:08.227064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" event={"ID":"221174c0-7107-451a-b26b-c09ba0f5fb74","Type":"ContainerStarted","Data":"f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724"}
Sep 17 18:56:08.227276 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:08.227157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" event={"ID":"221174c0-7107-451a-b26b-c09ba0f5fb74","Type":"ContainerStarted","Data":"6999f6317908528a2573b225797c92ab0336e7fbbb09441d1612752dfa336ac4"}
Sep 17 18:56:08.228795 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:08.228766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" event={"ID":"45b4773d-8793-4bd4-8f15-b074bf4a18cc","Type":"ContainerStarted","Data":"d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89"}
Sep 17 18:56:08.228936 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:08.228820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" event={"ID":"45b4773d-8793-4bd4-8f15-b074bf4a18cc","Type":"ContainerStarted","Data":"ab188f81c117115bd9559d51768642c7b38045e7f6decd7eeeb687e7d3fa5b7c"}
Sep 17 18:56:11.244350 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:11.244321    2577 generic.go:356] "Generic (PLEG): container finished" podID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerID="f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724" exitCode=0
Sep 17 18:56:11.244656 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:11.244376    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" event={"ID":"221174c0-7107-451a-b26b-c09ba0f5fb74","Type":"ContainerDied","Data":"f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724"}
Sep 17 18:56:12.251073 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:12.251030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" event={"ID":"221174c0-7107-451a-b26b-c09ba0f5fb74","Type":"ContainerStarted","Data":"d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152"}
Sep 17 18:56:12.251476 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:12.251100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" event={"ID":"221174c0-7107-451a-b26b-c09ba0f5fb74","Type":"ContainerStarted","Data":"c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e"}
Sep 17 18:56:12.251476 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:12.251289    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:12.252464 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:12.252443    2577 generic.go:356] "Generic (PLEG): container finished" podID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerID="d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89" exitCode=0
Sep 17 18:56:12.252562 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:12.252494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" event={"ID":"45b4773d-8793-4bd4-8f15-b074bf4a18cc","Type":"ContainerDied","Data":"d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89"}
Sep 17 18:56:12.270678 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:12.270623    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podStartSLOduration=2.404731456 podStartE2EDuration="6.270604301s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.865872845s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:12.269651204 +0000 UTC m=+1607.316054147" watchObservedRunningTime="2026-09-17 18:56:12.270604301 +0000 UTC m=+1607.317007206"
Sep 17 18:56:13.258529 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:13.258486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" event={"ID":"45b4773d-8793-4bd4-8f15-b074bf4a18cc","Type":"ContainerStarted","Data":"32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121"}
Sep 17 18:56:13.259006 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:13.258545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" event={"ID":"45b4773d-8793-4bd4-8f15-b074bf4a18cc","Type":"ContainerStarted","Data":"d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05"}
Sep 17 18:56:13.259006 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:13.258739    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:13.259814 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:13.259791    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:56:13.285297 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:13.285244    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podStartSLOduration=3.671366853 podStartE2EDuration="7.285226858s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.613860005s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:13.281786116 +0000 UTC m=+1608.328189043" watchObservedRunningTime="2026-09-17 18:56:13.285226858 +0000 UTC m=+1608.331629763"
Sep 17 18:56:14.262889 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:14.262841    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:56:18.259557 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:18.259522    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:18.260164 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:18.260003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:18.260985 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:18.260958    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:56:18.264930 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:18.264894    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:56:18.281083 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:18.281041    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:56:19.267623 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:19.267587    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:56:19.268248 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:19.268217    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:56:19.285543 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:19.285515    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:56:29.268438 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:29.268396    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:56:29.285914 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:29.285863    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:56:39.269187 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:39.269146    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:56:39.285895 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:39.285859    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:56:49.268321 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:49.268278    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:56:49.285983 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:49.285929    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:56:59.269056 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:59.269014    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:56:59.286162 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:56:59.286109    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:57:09.269089 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:09.269049    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:57:09.285966 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:09.285924    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:57:19.268583 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:19.268547    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:57:19.286115 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:19.286081    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:57:29.269082 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:29.269053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:57:29.286059 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:29.286030    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:57:29.481363 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:29.481329    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"]
Sep 17 18:57:29.610577 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:29.610475    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container" containerID="cri-o://c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e" gracePeriod=30
Sep 17 18:57:29.610577 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:29.610505    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kube-rbac-proxy" containerID="cri-o://d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152" gracePeriod=30
Sep 17 18:57:30.617044 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:30.617012    2577 generic.go:356] "Generic (PLEG): container finished" podID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerID="d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152" exitCode=2
Sep 17 18:57:30.617403 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:30.617051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" event={"ID":"221174c0-7107-451a-b26b-c09ba0f5fb74","Type":"ContainerDied","Data":"d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152"}
Sep 17 18:57:34.263356 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.263308    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 18:57:34.449626 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.449597    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:57:34.461499 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.461475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-467w5\" (UniqueName: \"kubernetes.io/projected/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-api-access-467w5\") pod \"221174c0-7107-451a-b26b-c09ba0f5fb74\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") "
Sep 17 18:57:34.461615 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.461545    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221174c0-7107-451a-b26b-c09ba0f5fb74-kserve-provision-location\") pod \"221174c0-7107-451a-b26b-c09ba0f5fb74\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") "
Sep 17 18:57:34.461663 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.461635    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-cabundle-cert\") pod \"221174c0-7107-451a-b26b-c09ba0f5fb74\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") "
Sep 17 18:57:34.461712 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.461672    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-rbac-proxy-sar-config\") pod \"221174c0-7107-451a-b26b-c09ba0f5fb74\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") "
Sep 17 18:57:34.461712 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.461698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221174c0-7107-451a-b26b-c09ba0f5fb74-proxy-tls\") pod \"221174c0-7107-451a-b26b-c09ba0f5fb74\" (UID: \"221174c0-7107-451a-b26b-c09ba0f5fb74\") "
Sep 17 18:57:34.461822 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.461796    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221174c0-7107-451a-b26b-c09ba0f5fb74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "221174c0-7107-451a-b26b-c09ba0f5fb74" (UID: "221174c0-7107-451a-b26b-c09ba0f5fb74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:34.462037 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.462014    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221174c0-7107-451a-b26b-c09ba0f5fb74-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.462129 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.462031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "221174c0-7107-451a-b26b-c09ba0f5fb74" (UID: "221174c0-7107-451a-b26b-c09ba0f5fb74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:34.462129 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.462043    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "221174c0-7107-451a-b26b-c09ba0f5fb74" (UID: "221174c0-7107-451a-b26b-c09ba0f5fb74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:34.463657 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.463636    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-api-access-467w5" (OuterVolumeSpecName: "kube-api-access-467w5") pod "221174c0-7107-451a-b26b-c09ba0f5fb74" (UID: "221174c0-7107-451a-b26b-c09ba0f5fb74"). InnerVolumeSpecName "kube-api-access-467w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:34.463732 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.463711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221174c0-7107-451a-b26b-c09ba0f5fb74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "221174c0-7107-451a-b26b-c09ba0f5fb74" (UID: "221174c0-7107-451a-b26b-c09ba0f5fb74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:34.563488 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.563398    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-467w5\" (UniqueName: \"kubernetes.io/projected/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-api-access-467w5\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.563488 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.563427    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.563488 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.563437    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221174c0-7107-451a-b26b-c09ba0f5fb74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.563488 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.563450    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221174c0-7107-451a-b26b-c09ba0f5fb74-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.637211 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.637175    2577 generic.go:356] "Generic (PLEG): container finished" podID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerID="c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e" exitCode=0
Sep 17 18:57:34.637361 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.637227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" event={"ID":"221174c0-7107-451a-b26b-c09ba0f5fb74","Type":"ContainerDied","Data":"c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e"}
Sep 17 18:57:34.637361 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.637244    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"
Sep 17 18:57:34.637361 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.637290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr" event={"ID":"221174c0-7107-451a-b26b-c09ba0f5fb74","Type":"ContainerDied","Data":"6999f6317908528a2573b225797c92ab0336e7fbbb09441d1612752dfa336ac4"}
Sep 17 18:57:34.637361 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.637327    2577 scope.go:117] "RemoveContainer" containerID="d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152"
Sep 17 18:57:34.646124 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.646108    2577 scope.go:117] "RemoveContainer" containerID="c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e"
Sep 17 18:57:34.654061 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.654008    2577 scope.go:117] "RemoveContainer" containerID="f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724"
Sep 17 18:57:34.661783 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.661760    2577 scope.go:117] "RemoveContainer" containerID="d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152"
Sep 17 18:57:34.662302 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:57:34.662133    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152\": container with ID starting with d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152 not found: ID does not exist" containerID="d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152"
Sep 17 18:57:34.662302 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.662178    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152"} err="failed to get container status \"d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152\": rpc error: code = NotFound desc = could not find container \"d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152\": container with ID starting with d1f639d5e89e0e5366cdc239e7328d982d5c74de61840b06e5fe0d9833da7152 not found: ID does not exist"
Sep 17 18:57:34.662302 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.662202    2577 scope.go:117] "RemoveContainer" containerID="c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e"
Sep 17 18:57:34.663933 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:57:34.663879    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e\": container with ID starting with c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e not found: ID does not exist" containerID="c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e"
Sep 17 18:57:34.664038 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.663932    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e"} err="failed to get container status \"c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e\": rpc error: code = NotFound desc = could not find container \"c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e\": container with ID starting with c888f6c61ec80b74320e4d6482989c02e57aaf52953eed70dc7b5dd2c40de07e not found: ID does not exist"
Sep 17 18:57:34.664038 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.663954    2577 scope.go:117] "RemoveContainer" containerID="f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724"
Sep 17 18:57:34.664250 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:57:34.664229    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724\": container with ID starting with f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724 not found: ID does not exist" containerID="f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724"
Sep 17 18:57:34.664297 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.664256    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724"} err="failed to get container status \"f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724\": rpc error: code = NotFound desc = could not find container \"f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724\": container with ID starting with f96a443bf9affd7fdbe593d7e46834158b471b36c66b65f02e4c7d9317836724 not found: ID does not exist"
Sep 17 18:57:34.664948 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.664927    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"]
Sep 17 18:57:34.677071 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:34.677045    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-ml2zr"]
Sep 17 18:57:35.239249 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:57:35.239216    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" path="/var/lib/kubelet/pods/221174c0-7107-451a-b26b-c09ba0f5fb74/volumes"
Sep 17 18:58:17.464374 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.464339    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"]
Sep 17 18:58:17.465045 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.464996    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="storage-initializer"
Sep 17 18:58:17.465045 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.465017    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="storage-initializer"
Sep 17 18:58:17.465045 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.465031    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kube-rbac-proxy"
Sep 17 18:58:17.465045 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.465040    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kube-rbac-proxy"
Sep 17 18:58:17.465291 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.465071    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container"
Sep 17 18:58:17.465291 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.465081    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container"
Sep 17 18:58:17.465291 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.465166    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kserve-container"
Sep 17 18:58:17.465291 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.465179    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="221174c0-7107-451a-b26b-c09ba0f5fb74" containerName="kube-rbac-proxy"
Sep 17 18:58:17.468627 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.468607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.471171 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.471153    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 18:58:17.471290 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.471274    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 18:58:17.485180 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.485157    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"]
Sep 17 18:58:17.530816 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.530791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b682269b-f3c9-4d35-98b4-3f783966cd7d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.530816 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.530816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tjzh\" (UniqueName: \"kubernetes.io/projected/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-api-access-8tjzh\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.530997 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.530844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b682269b-f3c9-4d35-98b4-3f783966cd7d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.530997 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.530954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.530997 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.530992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.631832 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.631795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.632010 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.631856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b682269b-f3c9-4d35-98b4-3f783966cd7d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.632010 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.631873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tjzh\" (UniqueName: \"kubernetes.io/projected/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-api-access-8tjzh\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.632010 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.631922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b682269b-f3c9-4d35-98b4-3f783966cd7d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.632010 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.631981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.632420 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.632396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b682269b-f3c9-4d35-98b4-3f783966cd7d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.632541 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.632521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.632607 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.632582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.634307 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.634288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b682269b-f3c9-4d35-98b4-3f783966cd7d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.641598 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.641565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tjzh\" (UniqueName: \"kubernetes.io/projected/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-api-access-8tjzh\") pod \"isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:17.778801 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:17.778736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:18.063537 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:18.063465    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"]
Sep 17 18:58:18.069262 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:58:18.069237    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb682269b_f3c9_4d35_98b4_3f783966cd7d.slice/crio-1738b8f5525da982328f01b739db8c995570765209e07cb86fb60ad2756479b3 WatchSource:0}: Error finding container 1738b8f5525da982328f01b739db8c995570765209e07cb86fb60ad2756479b3: Status 404 returned error can't find the container with id 1738b8f5525da982328f01b739db8c995570765209e07cb86fb60ad2756479b3
Sep 17 18:58:18.846241 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:18.846198    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" event={"ID":"b682269b-f3c9-4d35-98b4-3f783966cd7d","Type":"ContainerStarted","Data":"60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223"}
Sep 17 18:58:18.846659 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:18.846296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" event={"ID":"b682269b-f3c9-4d35-98b4-3f783966cd7d","Type":"ContainerStarted","Data":"1738b8f5525da982328f01b739db8c995570765209e07cb86fb60ad2756479b3"}
Sep 17 18:58:22.864997 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:22.864966    2577 generic.go:356] "Generic (PLEG): container finished" podID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerID="60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223" exitCode=0
Sep 17 18:58:22.865365 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:22.865042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" event={"ID":"b682269b-f3c9-4d35-98b4-3f783966cd7d","Type":"ContainerDied","Data":"60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223"}
Sep 17 18:58:23.872096 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:23.872060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" event={"ID":"b682269b-f3c9-4d35-98b4-3f783966cd7d","Type":"ContainerStarted","Data":"a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb"}
Sep 17 18:58:23.872505 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:23.872122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" event={"ID":"b682269b-f3c9-4d35-98b4-3f783966cd7d","Type":"ContainerStarted","Data":"2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823"}
Sep 17 18:58:23.872505 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:23.872316    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:23.893860 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:23.893807    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podStartSLOduration=3.06323395 podStartE2EDuration="6.893790205s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.830556255s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:23.891705718 +0000 UTC m=+1738.938108632" watchObservedRunningTime="2026-09-17 18:58:23.893790205 +0000 UTC m=+1738.940193107"
Sep 17 18:58:24.876617 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:24.876584    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:24.877870 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:24.877839    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:58:25.881877 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:25.881833    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:58:30.886971 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:30.886931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:58:30.887520 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:30.887491    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:58:40.887453 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:40.887410    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:58:50.887744 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:58:50.887695    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:59:00.888009 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:00.887969    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:59:06.778208 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:06.778173    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"]
Sep 17 18:59:06.778652 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:06.778455    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" containerID="cri-o://d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05" gracePeriod=30
Sep 17 18:59:06.778652 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:06.778462    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kube-rbac-proxy" containerID="cri-o://32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121" gracePeriod=30
Sep 17 18:59:07.085379 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:07.085284    2577 generic.go:356] "Generic (PLEG): container finished" podID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerID="32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121" exitCode=2
Sep 17 18:59:07.085553 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:07.085366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" event={"ID":"45b4773d-8793-4bd4-8f15-b074bf4a18cc","Type":"ContainerDied","Data":"32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121"}
Sep 17 18:59:08.265465 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:08.265375    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 18:59:09.285959 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:09.285897    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:59:10.888001 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:10.887962    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:59:11.417550 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.417526    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:59:11.506782 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.506711    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-rbac-proxy-sar-config\") pod \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") "
Sep 17 18:59:11.506782 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.506771    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-cabundle-cert\") pod \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") "
Sep 17 18:59:11.506982 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.506822    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kserve-provision-location\") pod \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") "
Sep 17 18:59:11.506982 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.506841    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4s86\" (UniqueName: \"kubernetes.io/projected/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-api-access-k4s86\") pod \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") "
Sep 17 18:59:11.506982 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.506876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4773d-8793-4bd4-8f15-b074bf4a18cc-proxy-tls\") pod \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\" (UID: \"45b4773d-8793-4bd4-8f15-b074bf4a18cc\") "
Sep 17 18:59:11.507185 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.507157    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45b4773d-8793-4bd4-8f15-b074bf4a18cc" (UID: "45b4773d-8793-4bd4-8f15-b074bf4a18cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:11.507231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.507187    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45b4773d-8793-4bd4-8f15-b074bf4a18cc" (UID: "45b4773d-8793-4bd4-8f15-b074bf4a18cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:11.507231 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.507203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45b4773d-8793-4bd4-8f15-b074bf4a18cc" (UID: "45b4773d-8793-4bd4-8f15-b074bf4a18cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:11.508969 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.508945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b4773d-8793-4bd4-8f15-b074bf4a18cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45b4773d-8793-4bd4-8f15-b074bf4a18cc" (UID: "45b4773d-8793-4bd4-8f15-b074bf4a18cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:11.508969 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.508948    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-api-access-k4s86" (OuterVolumeSpecName: "kube-api-access-k4s86") pod "45b4773d-8793-4bd4-8f15-b074bf4a18cc" (UID: "45b4773d-8793-4bd4-8f15-b074bf4a18cc"). InnerVolumeSpecName "kube-api-access-k4s86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:11.607606 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.607581    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:59:11.607606 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.607602    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4s86\" (UniqueName: \"kubernetes.io/projected/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-api-access-k4s86\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:59:11.607743 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.607612    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4773d-8793-4bd4-8f15-b074bf4a18cc-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:59:11.607743 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.607621    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:59:11.607743 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:11.607629    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b4773d-8793-4bd4-8f15-b074bf4a18cc-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:59:12.065462 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.065430    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"]
Sep 17 18:59:12.065835 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.065796    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container"
Sep 17 18:59:12.065835 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.065806    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container"
Sep 17 18:59:12.065835 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.065815    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kube-rbac-proxy"
Sep 17 18:59:12.065835 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.065820    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kube-rbac-proxy"
Sep 17 18:59:12.065835 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.065829    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="storage-initializer"
Sep 17 18:59:12.065835 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.065835    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="storage-initializer"
Sep 17 18:59:12.066080 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.065912    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kserve-container"
Sep 17 18:59:12.066080 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.065923    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerName="kube-rbac-proxy"
Sep 17 18:59:12.069357 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.069339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.071700 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.071680    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:59:12.072088 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.072071    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:59:12.081417 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.081395    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"]
Sep 17 18:59:12.110626 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.110601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.110776 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.110655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.110835 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.110781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8206ca19-ece4-4cef-ac97-b58904dc26fb-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.110835 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.110816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5hs\" (UniqueName: \"kubernetes.io/projected/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-api-access-qk5hs\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.110961 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.110926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8206ca19-ece4-4cef-ac97-b58904dc26fb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.111822 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.111800    2577 generic.go:356] "Generic (PLEG): container finished" podID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" containerID="d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05" exitCode=0
Sep 17 18:59:12.111881 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.111857    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"
Sep 17 18:59:12.111933 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.111883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" event={"ID":"45b4773d-8793-4bd4-8f15-b074bf4a18cc","Type":"ContainerDied","Data":"d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05"}
Sep 17 18:59:12.111974 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.111960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv" event={"ID":"45b4773d-8793-4bd4-8f15-b074bf4a18cc","Type":"ContainerDied","Data":"ab188f81c117115bd9559d51768642c7b38045e7f6decd7eeeb687e7d3fa5b7c"}
Sep 17 18:59:12.112011 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.112002    2577 scope.go:117] "RemoveContainer" containerID="32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121"
Sep 17 18:59:12.120751 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.120734    2577 scope.go:117] "RemoveContainer" containerID="d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05"
Sep 17 18:59:12.128056 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.128040    2577 scope.go:117] "RemoveContainer" containerID="d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89"
Sep 17 18:59:12.133719 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.133700    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"]
Sep 17 18:59:12.136002 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.135976    2577 scope.go:117] "RemoveContainer" containerID="32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121"
Sep 17 18:59:12.136238 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:59:12.136219    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121\": container with ID starting with 32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121 not found: ID does not exist" containerID="32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121"
Sep 17 18:59:12.136310 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.136278    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121"} err="failed to get container status \"32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121\": rpc error: code = NotFound desc = could not find container \"32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121\": container with ID starting with 32f16f87fa37d99e9506ccfdc626bad189975a123547d9399bbf774056ea0121 not found: ID does not exist"
Sep 17 18:59:12.136310 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.136302    2577 scope.go:117] "RemoveContainer" containerID="d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05"
Sep 17 18:59:12.136525 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:59:12.136510    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05\": container with ID starting with d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05 not found: ID does not exist" containerID="d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05"
Sep 17 18:59:12.136571 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.136530    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05"} err="failed to get container status \"d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05\": rpc error: code = NotFound desc = could not find container \"d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05\": container with ID starting with d45a4aa277559c4aa26675c38d3a9b26f908686770750eb356fbcd934ddf2b05 not found: ID does not exist"
Sep 17 18:59:12.136571 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.136544    2577 scope.go:117] "RemoveContainer" containerID="d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89"
Sep 17 18:59:12.136776 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:59:12.136754    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89\": container with ID starting with d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89 not found: ID does not exist" containerID="d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89"
Sep 17 18:59:12.136825 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.136783    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89"} err="failed to get container status \"d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89\": rpc error: code = NotFound desc = could not find container \"d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89\": container with ID starting with d360158a0ae6cdca96f173abcfb53a1e3ba7f77c1bddc649bb2f68d68203bb89 not found: ID does not exist"
Sep 17 18:59:12.137434 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.137413    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-tpnlv"]
Sep 17 18:59:12.211679 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.211650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.211815 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.211719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8206ca19-ece4-4cef-ac97-b58904dc26fb-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.211815 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.211739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5hs\" (UniqueName: \"kubernetes.io/projected/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-api-access-qk5hs\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.211815 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.211764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8206ca19-ece4-4cef-ac97-b58904dc26fb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.211973 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.211811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.211973 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:59:12.211874    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:59:12.211973 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:59:12.211964    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8206ca19-ece4-4cef-ac97-b58904dc26fb-proxy-tls podName:8206ca19-ece4-4cef-ac97-b58904dc26fb nodeName:}" failed. No retries permitted until 2026-09-17 18:59:12.711942678 +0000 UTC m=+1787.758345571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8206ca19-ece4-4cef-ac97-b58904dc26fb-proxy-tls") pod "isvc-canary-rollback-predictor-7679bbb844-qf95h" (UID: "8206ca19-ece4-4cef-ac97-b58904dc26fb") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:59:12.212124 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.212110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8206ca19-ece4-4cef-ac97-b58904dc26fb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.212350 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.212331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.212386 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.212374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.220674 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.220655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5hs\" (UniqueName: \"kubernetes.io/projected/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-api-access-qk5hs\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.716947 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.716888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8206ca19-ece4-4cef-ac97-b58904dc26fb-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.719242 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.719223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8206ca19-ece4-4cef-ac97-b58904dc26fb-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-qf95h\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:12.978757 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:12.978674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:13.239128 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:13.239044    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b4773d-8793-4bd4-8f15-b074bf4a18cc" path="/var/lib/kubelet/pods/45b4773d-8793-4bd4-8f15-b074bf4a18cc/volumes"
Sep 17 18:59:13.253431 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:13.253410    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"]
Sep 17 18:59:13.255328 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:59:13.255305    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8206ca19_ece4_4cef_ac97_b58904dc26fb.slice/crio-64b41ffd663f89f439f192a947194b7e369368520a3649f23aa165611fd66ba2 WatchSource:0}: Error finding container 64b41ffd663f89f439f192a947194b7e369368520a3649f23aa165611fd66ba2: Status 404 returned error can't find the container with id 64b41ffd663f89f439f192a947194b7e369368520a3649f23aa165611fd66ba2
Sep 17 18:59:14.124556 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:14.124510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" event={"ID":"8206ca19-ece4-4cef-ac97-b58904dc26fb","Type":"ContainerStarted","Data":"e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d"}
Sep 17 18:59:14.124755 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:14.124604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" event={"ID":"8206ca19-ece4-4cef-ac97-b58904dc26fb","Type":"ContainerStarted","Data":"64b41ffd663f89f439f192a947194b7e369368520a3649f23aa165611fd66ba2"}
Sep 17 18:59:18.143720 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:18.143633    2577 generic.go:356] "Generic (PLEG): container finished" podID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerID="e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d" exitCode=0
Sep 17 18:59:18.144172 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:18.143711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" event={"ID":"8206ca19-ece4-4cef-ac97-b58904dc26fb","Type":"ContainerDied","Data":"e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d"}
Sep 17 18:59:19.150212 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:19.150162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" event={"ID":"8206ca19-ece4-4cef-ac97-b58904dc26fb","Type":"ContainerStarted","Data":"7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809"}
Sep 17 18:59:19.150708 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:19.150251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" event={"ID":"8206ca19-ece4-4cef-ac97-b58904dc26fb","Type":"ContainerStarted","Data":"6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c"}
Sep 17 18:59:19.150708 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:19.150473    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:19.150708 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:19.150530    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:19.151842 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:19.151818    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:59:19.170705 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:19.170664    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podStartSLOduration=3.098296212 podStartE2EDuration="7.170649315s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.072353103s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:59:19.169048816 +0000 UTC m=+1794.215451732" watchObservedRunningTime="2026-09-17 18:59:19.170649315 +0000 UTC m=+1794.217052216"
Sep 17 18:59:20.154801 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:20.154756    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:59:20.888101 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:20.888063    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:59:25.160038 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:25.160010    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 18:59:25.160561 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:25.160534    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:59:30.887741 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:30.887697    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:59:35.160692 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:35.160656    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:59:40.888495 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:40.888457    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:59:45.160779 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:45.160739    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:59:47.593565 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:47.593529    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"]
Sep 17 18:59:47.593935 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:47.593754    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" containerID="cri-o://2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823" gracePeriod=30
Sep 17 18:59:47.593935 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:47.593786    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kube-rbac-proxy" containerID="cri-o://a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb" gracePeriod=30
Sep 17 18:59:48.290094 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:48.290058    2577 generic.go:356] "Generic (PLEG): container finished" podID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerID="a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb" exitCode=2
Sep 17 18:59:48.290310 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:48.290140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" event={"ID":"b682269b-f3c9-4d35-98b4-3f783966cd7d","Type":"ContainerDied","Data":"a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb"}
Sep 17 18:59:50.882746 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:50.882704    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 17 18:59:50.888036 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:50.888003    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:59:52.247923 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.247887    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:59:52.309479 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.309451    2577 generic.go:356] "Generic (PLEG): container finished" podID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerID="2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823" exitCode=0
Sep 17 18:59:52.309611 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.309503    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"
Sep 17 18:59:52.309611 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.309531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" event={"ID":"b682269b-f3c9-4d35-98b4-3f783966cd7d","Type":"ContainerDied","Data":"2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823"}
Sep 17 18:59:52.309611 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.309590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw" event={"ID":"b682269b-f3c9-4d35-98b4-3f783966cd7d","Type":"ContainerDied","Data":"1738b8f5525da982328f01b739db8c995570765209e07cb86fb60ad2756479b3"}
Sep 17 18:59:52.309738 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.309617    2577 scope.go:117] "RemoveContainer" containerID="a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb"
Sep 17 18:59:52.317866 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.317850    2577 scope.go:117] "RemoveContainer" containerID="2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823"
Sep 17 18:59:52.325201 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.325185    2577 scope.go:117] "RemoveContainer" containerID="60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223"
Sep 17 18:59:52.334993 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.334981    2577 scope.go:117] "RemoveContainer" containerID="a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb"
Sep 17 18:59:52.335265 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:59:52.335249    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb\": container with ID starting with a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb not found: ID does not exist" containerID="a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb"
Sep 17 18:59:52.335315 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.335273    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb"} err="failed to get container status \"a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb\": rpc error: code = NotFound desc = could not find container \"a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb\": container with ID starting with a82139169d5b2ae9aed85f81a6c7edfccf024df598ba9a2d3376e9984b2124bb not found: ID does not exist"
Sep 17 18:59:52.335315 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.335288    2577 scope.go:117] "RemoveContainer" containerID="2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823"
Sep 17 18:59:52.335541 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:59:52.335520    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823\": container with ID starting with 2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823 not found: ID does not exist" containerID="2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823"
Sep 17 18:59:52.335578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.335551    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823"} err="failed to get container status \"2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823\": rpc error: code = NotFound desc = could not find container \"2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823\": container with ID starting with 2d56e85b755ff1a387e726e81265f61b017c0223e364b210cc4a20df078d4823 not found: ID does not exist"
Sep 17 18:59:52.335578 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.335573    2577 scope.go:117] "RemoveContainer" containerID="60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223"
Sep 17 18:59:52.335849 ip-10-0-131-128 kubenswrapper[2577]: E0917 18:59:52.335832    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223\": container with ID starting with 60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223 not found: ID does not exist" containerID="60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223"
Sep 17 18:59:52.335891 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.335862    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223"} err="failed to get container status \"60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223\": rpc error: code = NotFound desc = could not find container \"60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223\": container with ID starting with 60ee49d8249c6cb985dde8478accbc5086ffc3b77adda0e50073c6a576735223 not found: ID does not exist"
Sep 17 18:59:52.347619 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.347600    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b682269b-f3c9-4d35-98b4-3f783966cd7d-kserve-provision-location\") pod \"b682269b-f3c9-4d35-98b4-3f783966cd7d\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") "
Sep 17 18:59:52.347703 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.347653    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-rbac-proxy-sar-config\") pod \"b682269b-f3c9-4d35-98b4-3f783966cd7d\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") "
Sep 17 18:59:52.347766 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.347701    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-cabundle-cert\") pod \"b682269b-f3c9-4d35-98b4-3f783966cd7d\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") "
Sep 17 18:59:52.347766 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.347744    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tjzh\" (UniqueName: \"kubernetes.io/projected/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-api-access-8tjzh\") pod \"b682269b-f3c9-4d35-98b4-3f783966cd7d\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") "
Sep 17 18:59:52.347865 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.347780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b682269b-f3c9-4d35-98b4-3f783966cd7d-proxy-tls\") pod \"b682269b-f3c9-4d35-98b4-3f783966cd7d\" (UID: \"b682269b-f3c9-4d35-98b4-3f783966cd7d\") "
Sep 17 18:59:52.347931 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.347868    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b682269b-f3c9-4d35-98b4-3f783966cd7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b682269b-f3c9-4d35-98b4-3f783966cd7d" (UID: "b682269b-f3c9-4d35-98b4-3f783966cd7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:52.348063 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.348041    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b682269b-f3c9-4d35-98b4-3f783966cd7d" (UID: "b682269b-f3c9-4d35-98b4-3f783966cd7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:52.348108 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.348081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b682269b-f3c9-4d35-98b4-3f783966cd7d" (UID: "b682269b-f3c9-4d35-98b4-3f783966cd7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:52.348179 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.348167    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b682269b-f3c9-4d35-98b4-3f783966cd7d-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:59:52.348218 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.348183    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:59:52.348218 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.348193    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b682269b-f3c9-4d35-98b4-3f783966cd7d-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:59:52.349676 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.349657    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-api-access-8tjzh" (OuterVolumeSpecName: "kube-api-access-8tjzh") pod "b682269b-f3c9-4d35-98b4-3f783966cd7d" (UID: "b682269b-f3c9-4d35-98b4-3f783966cd7d"). InnerVolumeSpecName "kube-api-access-8tjzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:52.349727 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.349687    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b682269b-f3c9-4d35-98b4-3f783966cd7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b682269b-f3c9-4d35-98b4-3f783966cd7d" (UID: "b682269b-f3c9-4d35-98b4-3f783966cd7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:52.448703 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.448680    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tjzh\" (UniqueName: \"kubernetes.io/projected/b682269b-f3c9-4d35-98b4-3f783966cd7d-kube-api-access-8tjzh\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:59:52.448703 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.448702    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b682269b-f3c9-4d35-98b4-3f783966cd7d-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 18:59:52.631227 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.631196    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"]
Sep 17 18:59:52.636335 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.636311    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66cff85cb4-tzppw"]
Sep 17 18:59:52.789144 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.789114    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"]
Sep 17 18:59:52.789511 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.789497    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kube-rbac-proxy"
Sep 17 18:59:52.789556 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.789513    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kube-rbac-proxy"
Sep 17 18:59:52.789556 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.789529    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="storage-initializer"
Sep 17 18:59:52.789556 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.789536    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="storage-initializer"
Sep 17 18:59:52.789556 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.789547    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container"
Sep 17 18:59:52.789556 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.789553    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container"
Sep 17 18:59:52.789712 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.789605    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kube-rbac-proxy"
Sep 17 18:59:52.789712 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.789614    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" containerName="kserve-container"
Sep 17 18:59:52.794365 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.794344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:52.796644 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.796625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 18:59:52.796744 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.796711    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 18:59:52.802982 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.802961    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"]
Sep 17 18:59:52.952846 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.952805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:52.953047 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.952864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgpkx\" (UniqueName: \"kubernetes.io/projected/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-api-access-bgpkx\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:52.953047 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.952987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:52.953047 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.953030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9cf6209-1754-4007-ad76-44e65c5af89a-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:52.953203 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:52.953085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9cf6209-1754-4007-ad76-44e65c5af89a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.054466 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.054435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.054638 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.054479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgpkx\" (UniqueName: \"kubernetes.io/projected/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-api-access-bgpkx\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.054638 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.054519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.054638 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.054541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9cf6209-1754-4007-ad76-44e65c5af89a-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.054638 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.054571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9cf6209-1754-4007-ad76-44e65c5af89a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.055025 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.054998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9cf6209-1754-4007-ad76-44e65c5af89a-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.055238 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.055220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.055238 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.055228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.056865 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.056849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9cf6209-1754-4007-ad76-44e65c5af89a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.063184 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.063166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgpkx\" (UniqueName: \"kubernetes.io/projected/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-api-access-bgpkx\") pod \"isvc-tensorflow-predictor-547dfc5669-sgqgq\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.105221 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.105194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 18:59:53.238710 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.238633    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b682269b-f3c9-4d35-98b4-3f783966cd7d" path="/var/lib/kubelet/pods/b682269b-f3c9-4d35-98b4-3f783966cd7d/volumes"
Sep 17 18:59:53.376302 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:53.376268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"]
Sep 17 18:59:53.378854 ip-10-0-131-128 kubenswrapper[2577]: W0917 18:59:53.378825    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cf6209_1754_4007_ad76_44e65c5af89a.slice/crio-9bc54c566043cd46d249ed7d85a96a447558c2e504e49326fc9bf1c86d26112f WatchSource:0}: Error finding container 9bc54c566043cd46d249ed7d85a96a447558c2e504e49326fc9bf1c86d26112f: Status 404 returned error can't find the container with id 9bc54c566043cd46d249ed7d85a96a447558c2e504e49326fc9bf1c86d26112f
Sep 17 18:59:54.321880 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:54.321837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" event={"ID":"a9cf6209-1754-4007-ad76-44e65c5af89a","Type":"ContainerStarted","Data":"81da37f520acab87f44ebf7bc507d2f91e56fa8d6bcdae036d8832a5c4800530"}
Sep 17 18:59:54.322106 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:54.321961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" event={"ID":"a9cf6209-1754-4007-ad76-44e65c5af89a","Type":"ContainerStarted","Data":"9bc54c566043cd46d249ed7d85a96a447558c2e504e49326fc9bf1c86d26112f"}
Sep 17 18:59:55.161064 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:55.161023    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:59:58.342268 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:58.342188    2577 generic.go:356] "Generic (PLEG): container finished" podID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerID="81da37f520acab87f44ebf7bc507d2f91e56fa8d6bcdae036d8832a5c4800530" exitCode=0
Sep 17 18:59:58.342268 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:58.342261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" event={"ID":"a9cf6209-1754-4007-ad76-44e65c5af89a","Type":"ContainerDied","Data":"81da37f520acab87f44ebf7bc507d2f91e56fa8d6bcdae036d8832a5c4800530"}
Sep 17 18:59:58.343448 ip-10-0-131-128 kubenswrapper[2577]: I0917 18:59:58.343434    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:00:02.368411 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:02.368375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" event={"ID":"a9cf6209-1754-4007-ad76-44e65c5af89a","Type":"ContainerStarted","Data":"a87bf59d022bd486708dec7e12377eff011b2f6a0487d48ac176fcc507b1d8c8"}
Sep 17 19:00:02.368783 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:02.368443    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" event={"ID":"a9cf6209-1754-4007-ad76-44e65c5af89a","Type":"ContainerStarted","Data":"504c2ff052746ee1ed574daae9e9584bb4a9512d54c6e254a83302d4565d2efa"}
Sep 17 19:00:02.368783 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:02.368636    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 19:00:02.391775 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:02.391721    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podStartSLOduration=2.252148925 podStartE2EDuration="10.391703462s" totalImagesPullingTime="3.53028468s" totalInitContainerRuntime="4.609269857s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:00:02.390577573 +0000 UTC m=+1837.436980494" watchObservedRunningTime="2026-09-17 19:00:02.391703462 +0000 UTC m=+1837.438106364"
Sep 17 19:00:03.373582 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:03.373553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 19:00:03.375070 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:03.375034    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:00:04.379395 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:04.379351    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:00:05.161023 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:05.160988    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:00:09.384088 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:09.384058    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 19:00:09.384599 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:09.384565    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:00:15.161299 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:15.161258    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:00:19.385873 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:19.385839    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 19:00:25.161351 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:25.161303    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:00:33.239446 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:33.239415    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"]
Sep 17 19:00:33.239872 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:33.239637    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kserve-container" containerID="cri-o://504c2ff052746ee1ed574daae9e9584bb4a9512d54c6e254a83302d4565d2efa" gracePeriod=30
Sep 17 19:00:33.239872 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:33.239666    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kube-rbac-proxy" containerID="cri-o://a87bf59d022bd486708dec7e12377eff011b2f6a0487d48ac176fcc507b1d8c8" gracePeriod=30
Sep 17 19:00:33.528166 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:33.528081    2577 generic.go:356] "Generic (PLEG): container finished" podID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerID="a87bf59d022bd486708dec7e12377eff011b2f6a0487d48ac176fcc507b1d8c8" exitCode=2
Sep 17 19:00:33.528166 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:33.528154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" event={"ID":"a9cf6209-1754-4007-ad76-44e65c5af89a","Type":"ContainerDied","Data":"a87bf59d022bd486708dec7e12377eff011b2f6a0487d48ac176fcc507b1d8c8"}
Sep 17 19:00:34.379986 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:34.379942    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:00:35.162131 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:35.162096    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 19:00:35.241497 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:35.241466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"]
Sep 17 19:00:35.538306 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:35.538268    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kube-rbac-proxy" containerID="cri-o://7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809" gracePeriod=30
Sep 17 19:00:35.538803 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:35.538244    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container" containerID="cri-o://6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c" gracePeriod=30
Sep 17 19:00:36.545305 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:36.545266    2577 generic.go:356] "Generic (PLEG): container finished" podID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerID="7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809" exitCode=2
Sep 17 19:00:36.545679 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:36.545313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" event={"ID":"8206ca19-ece4-4cef-ac97-b58904dc26fb","Type":"ContainerDied","Data":"7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809"}
Sep 17 19:00:39.380233 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:39.380136    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:00:40.155817 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.155765    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 19:00:40.477311 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.477283    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 19:00:40.566120 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.566087    2577 generic.go:356] "Generic (PLEG): container finished" podID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerID="6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c" exitCode=0
Sep 17 19:00:40.566318 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.566151    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"
Sep 17 19:00:40.566318 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.566172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" event={"ID":"8206ca19-ece4-4cef-ac97-b58904dc26fb","Type":"ContainerDied","Data":"6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c"}
Sep 17 19:00:40.566318 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.566237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h" event={"ID":"8206ca19-ece4-4cef-ac97-b58904dc26fb","Type":"ContainerDied","Data":"64b41ffd663f89f439f192a947194b7e369368520a3649f23aa165611fd66ba2"}
Sep 17 19:00:40.566318 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.566298    2577 scope.go:117] "RemoveContainer" containerID="7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809"
Sep 17 19:00:40.574550 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.574526    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-cabundle-cert\") pod \"8206ca19-ece4-4cef-ac97-b58904dc26fb\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") "
Sep 17 19:00:40.574686 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.574608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-rbac-proxy-sar-config\") pod \"8206ca19-ece4-4cef-ac97-b58904dc26fb\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") "
Sep 17 19:00:40.574686 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.574635    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8206ca19-ece4-4cef-ac97-b58904dc26fb-proxy-tls\") pod \"8206ca19-ece4-4cef-ac97-b58904dc26fb\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") "
Sep 17 19:00:40.574686 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.574651    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk5hs\" (UniqueName: \"kubernetes.io/projected/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-api-access-qk5hs\") pod \"8206ca19-ece4-4cef-ac97-b58904dc26fb\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") "
Sep 17 19:00:40.574686 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.574682    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8206ca19-ece4-4cef-ac97-b58904dc26fb-kserve-provision-location\") pod \"8206ca19-ece4-4cef-ac97-b58904dc26fb\" (UID: \"8206ca19-ece4-4cef-ac97-b58904dc26fb\") "
Sep 17 19:00:40.574993 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.574965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8206ca19-ece4-4cef-ac97-b58904dc26fb" (UID: "8206ca19-ece4-4cef-ac97-b58904dc26fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:00:40.575105 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.575082    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8206ca19-ece4-4cef-ac97-b58904dc26fb" (UID: "8206ca19-ece4-4cef-ac97-b58904dc26fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:00:40.575219 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.575194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8206ca19-ece4-4cef-ac97-b58904dc26fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8206ca19-ece4-4cef-ac97-b58904dc26fb" (UID: "8206ca19-ece4-4cef-ac97-b58904dc26fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:40.576813 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.576790    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-api-access-qk5hs" (OuterVolumeSpecName: "kube-api-access-qk5hs") pod "8206ca19-ece4-4cef-ac97-b58904dc26fb" (UID: "8206ca19-ece4-4cef-ac97-b58904dc26fb"). InnerVolumeSpecName "kube-api-access-qk5hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:00:40.576813 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.576798    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8206ca19-ece4-4cef-ac97-b58904dc26fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8206ca19-ece4-4cef-ac97-b58904dc26fb" (UID: "8206ca19-ece4-4cef-ac97-b58904dc26fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:00:40.592884 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.592857    2577 scope.go:117] "RemoveContainer" containerID="6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c"
Sep 17 19:00:40.600338 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.600324    2577 scope.go:117] "RemoveContainer" containerID="e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d"
Sep 17 19:00:40.607665 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.607648    2577 scope.go:117] "RemoveContainer" containerID="7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809"
Sep 17 19:00:40.607953 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:00:40.607928    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809\": container with ID starting with 7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809 not found: ID does not exist" containerID="7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809"
Sep 17 19:00:40.608025 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.607960    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809"} err="failed to get container status \"7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809\": rpc error: code = NotFound desc = could not find container \"7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809\": container with ID starting with 7c69c3db10bd73a1b5df23a0e20323e2b3f4162b09744f4fc9024f2bc827d809 not found: ID does not exist"
Sep 17 19:00:40.608025 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.607981    2577 scope.go:117] "RemoveContainer" containerID="6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c"
Sep 17 19:00:40.608251 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:00:40.608233    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c\": container with ID starting with 6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c not found: ID does not exist" containerID="6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c"
Sep 17 19:00:40.608297 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.608256    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c"} err="failed to get container status \"6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c\": rpc error: code = NotFound desc = could not find container \"6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c\": container with ID starting with 6069924ccf8e137b096a733b487717264c34ff5b8af0a8ca2bba860121aca96c not found: ID does not exist"
Sep 17 19:00:40.608297 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.608270    2577 scope.go:117] "RemoveContainer" containerID="e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d"
Sep 17 19:00:40.608466 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:00:40.608444    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d\": container with ID starting with e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d not found: ID does not exist" containerID="e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d"
Sep 17 19:00:40.608511 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.608469    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d"} err="failed to get container status \"e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d\": rpc error: code = NotFound desc = could not find container \"e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d\": container with ID starting with e122d894c6efc20b8eb7663eb2f3e3774dbaf410b1d3e2d1cb120290a017d21d not found: ID does not exist"
Sep 17 19:00:40.676058 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.676028    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.676058 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.676054    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.676058 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.676064    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8206ca19-ece4-4cef-ac97-b58904dc26fb-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.676268 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.676075    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk5hs\" (UniqueName: \"kubernetes.io/projected/8206ca19-ece4-4cef-ac97-b58904dc26fb-kube-api-access-qk5hs\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.676268 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.676088    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8206ca19-ece4-4cef-ac97-b58904dc26fb-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.890925 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.890875    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"]
Sep 17 19:00:40.895308 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:40.895280    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-qf95h"]
Sep 17 19:00:41.239084 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:41.239052    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" path="/var/lib/kubelet/pods/8206ca19-ece4-4cef-ac97-b58904dc26fb/volumes"
Sep 17 19:00:42.031404 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.031371    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"]
Sep 17 19:00:42.031803 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.031743    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="storage-initializer"
Sep 17 19:00:42.031803 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.031753    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="storage-initializer"
Sep 17 19:00:42.031803 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.031767    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kube-rbac-proxy"
Sep 17 19:00:42.031803 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.031772    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kube-rbac-proxy"
Sep 17 19:00:42.031803 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.031781    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container"
Sep 17 19:00:42.031803 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.031786    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container"
Sep 17 19:00:42.032030 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.031838    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kserve-container"
Sep 17 19:00:42.032030 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.031851    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8206ca19-ece4-4cef-ac97-b58904dc26fb" containerName="kube-rbac-proxy"
Sep 17 19:00:42.036560 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.036540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.039779 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.039758    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 19:00:42.039888 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.039795    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 19:00:42.047852 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.047824    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"]
Sep 17 19:00:42.188759 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.188727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396329aa-884b-4a4f-8095-34081d02c86d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.188964 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.188778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqf8b\" (UniqueName: \"kubernetes.io/projected/396329aa-884b-4a4f-8095-34081d02c86d-kube-api-access-pqf8b\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.188964 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.188815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.188964 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.188868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.189083 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.188961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396329aa-884b-4a4f-8095-34081d02c86d-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.290146 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.290045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqf8b\" (UniqueName: \"kubernetes.io/projected/396329aa-884b-4a4f-8095-34081d02c86d-kube-api-access-pqf8b\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.290146 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.290104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.290363 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.290166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.290363 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.290217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396329aa-884b-4a4f-8095-34081d02c86d-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.290363 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.290260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396329aa-884b-4a4f-8095-34081d02c86d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.290657 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.290638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396329aa-884b-4a4f-8095-34081d02c86d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.290843 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.290825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.290898 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.290843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.292733 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.292711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396329aa-884b-4a4f-8095-34081d02c86d-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.302983 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.302960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqf8b\" (UniqueName: \"kubernetes.io/projected/396329aa-884b-4a4f-8095-34081d02c86d-kube-api-access-pqf8b\") pod \"isvc-canary-rollback-predictor-7679bbb844-kwwrc\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.347431 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.347398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:42.620367 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:42.620339    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"]
Sep 17 19:00:42.622780 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:00:42.622749    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396329aa_884b_4a4f_8095_34081d02c86d.slice/crio-8be0538e29a11598d2be17345f3c1353701f58bb6d34bd8c2a74e2a169e528c1 WatchSource:0}: Error finding container 8be0538e29a11598d2be17345f3c1353701f58bb6d34bd8c2a74e2a169e528c1: Status 404 returned error can't find the container with id 8be0538e29a11598d2be17345f3c1353701f58bb6d34bd8c2a74e2a169e528c1
Sep 17 19:00:43.584254 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:43.584216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" event={"ID":"396329aa-884b-4a4f-8095-34081d02c86d","Type":"ContainerStarted","Data":"ccac3ddb650eb332857cdb9b07590abd4c978777671c5fda075d7bd8d1ce6ca3"}
Sep 17 19:00:43.584654 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:43.584276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" event={"ID":"396329aa-884b-4a4f-8095-34081d02c86d","Type":"ContainerStarted","Data":"8be0538e29a11598d2be17345f3c1353701f58bb6d34bd8c2a74e2a169e528c1"}
Sep 17 19:00:44.380204 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:44.380161    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:00:44.380403 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:44.380281    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 19:00:46.600297 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:46.600262    2577 generic.go:356] "Generic (PLEG): container finished" podID="396329aa-884b-4a4f-8095-34081d02c86d" containerID="ccac3ddb650eb332857cdb9b07590abd4c978777671c5fda075d7bd8d1ce6ca3" exitCode=0
Sep 17 19:00:46.600669 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:46.600321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" event={"ID":"396329aa-884b-4a4f-8095-34081d02c86d","Type":"ContainerDied","Data":"ccac3ddb650eb332857cdb9b07590abd4c978777671c5fda075d7bd8d1ce6ca3"}
Sep 17 19:00:47.607340 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:47.607297    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" event={"ID":"396329aa-884b-4a4f-8095-34081d02c86d","Type":"ContainerStarted","Data":"ff505210b26795de988452298b6ad84d2606cb03d89f9be50d2d3eb0cc66ac74"}
Sep 17 19:00:47.607714 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:47.607369    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" event={"ID":"396329aa-884b-4a4f-8095-34081d02c86d","Type":"ContainerStarted","Data":"b3123d58c03c6400ec4fc2c704c49e5f23ad54482bbe3b7ad24b9cb81cf5bf11"}
Sep 17 19:00:47.607714 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:47.607617    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:47.625857 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:47.625800    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podStartSLOduration=1.874431274 podStartE2EDuration="5.625780909s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.751349635s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:00:47.624886271 +0000 UTC m=+1882.671289173" watchObservedRunningTime="2026-09-17 19:00:47.625780909 +0000 UTC m=+1882.672183814"
Sep 17 19:00:48.612512 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:48.612478    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:48.613882 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:48.613853    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:00:49.380301 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:49.380256    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:00:49.624724 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:49.624682    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:00:54.379522 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:54.379480    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:00:54.630039 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:54.629959    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:00:54.630667 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:54.630626    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:00:59.379508 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:00:59.379459    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:01:03.691166 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.691135    2577 generic.go:356] "Generic (PLEG): container finished" podID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerID="504c2ff052746ee1ed574daae9e9584bb4a9512d54c6e254a83302d4565d2efa" exitCode=137
Sep 17 19:01:03.691575 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.691212    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" event={"ID":"a9cf6209-1754-4007-ad76-44e65c5af89a","Type":"ContainerDied","Data":"504c2ff052746ee1ed574daae9e9584bb4a9512d54c6e254a83302d4565d2efa"}
Sep 17 19:01:03.886170 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.886134    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 19:01:03.966825 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.966739    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgpkx\" (UniqueName: \"kubernetes.io/projected/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-api-access-bgpkx\") pod \"a9cf6209-1754-4007-ad76-44e65c5af89a\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") "
Sep 17 19:01:03.967007 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.966834    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9cf6209-1754-4007-ad76-44e65c5af89a-proxy-tls\") pod \"a9cf6209-1754-4007-ad76-44e65c5af89a\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") "
Sep 17 19:01:03.967007 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.966862    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-cabundle-cert\") pod \"a9cf6209-1754-4007-ad76-44e65c5af89a\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") "
Sep 17 19:01:03.967007 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.966896    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-rbac-proxy-sar-config\") pod \"a9cf6209-1754-4007-ad76-44e65c5af89a\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") "
Sep 17 19:01:03.967007 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.966948    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9cf6209-1754-4007-ad76-44e65c5af89a-kserve-provision-location\") pod \"a9cf6209-1754-4007-ad76-44e65c5af89a\" (UID: \"a9cf6209-1754-4007-ad76-44e65c5af89a\") "
Sep 17 19:01:03.967302 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.967277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9cf6209-1754-4007-ad76-44e65c5af89a" (UID: "a9cf6209-1754-4007-ad76-44e65c5af89a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:01:03.967375 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.967334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9cf6209-1754-4007-ad76-44e65c5af89a" (UID: "a9cf6209-1754-4007-ad76-44e65c5af89a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:01:03.969017 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.968994    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-api-access-bgpkx" (OuterVolumeSpecName: "kube-api-access-bgpkx") pod "a9cf6209-1754-4007-ad76-44e65c5af89a" (UID: "a9cf6209-1754-4007-ad76-44e65c5af89a"). InnerVolumeSpecName "kube-api-access-bgpkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:03.969117 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.969038    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cf6209-1754-4007-ad76-44e65c5af89a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9cf6209-1754-4007-ad76-44e65c5af89a" (UID: "a9cf6209-1754-4007-ad76-44e65c5af89a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:03.977455 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:03.977429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cf6209-1754-4007-ad76-44e65c5af89a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9cf6209-1754-4007-ad76-44e65c5af89a" (UID: "a9cf6209-1754-4007-ad76-44e65c5af89a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:04.068198 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.068160    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:01:04.068198 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.068190    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9cf6209-1754-4007-ad76-44e65c5af89a-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:01:04.068198 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.068200    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgpkx\" (UniqueName: \"kubernetes.io/projected/a9cf6209-1754-4007-ad76-44e65c5af89a-kube-api-access-bgpkx\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:01:04.068198 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.068209    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9cf6209-1754-4007-ad76-44e65c5af89a-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:01:04.068483 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.068217    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9cf6209-1754-4007-ad76-44e65c5af89a-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:01:04.631313 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.631272    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:01:04.698023 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.697986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq" event={"ID":"a9cf6209-1754-4007-ad76-44e65c5af89a","Type":"ContainerDied","Data":"9bc54c566043cd46d249ed7d85a96a447558c2e504e49326fc9bf1c86d26112f"}
Sep 17 19:01:04.698023 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.698013    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"
Sep 17 19:01:04.698496 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.698065    2577 scope.go:117] "RemoveContainer" containerID="a87bf59d022bd486708dec7e12377eff011b2f6a0487d48ac176fcc507b1d8c8"
Sep 17 19:01:04.707067 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.707051    2577 scope.go:117] "RemoveContainer" containerID="504c2ff052746ee1ed574daae9e9584bb4a9512d54c6e254a83302d4565d2efa"
Sep 17 19:01:04.714569 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.714550    2577 scope.go:117] "RemoveContainer" containerID="81da37f520acab87f44ebf7bc507d2f91e56fa8d6bcdae036d8832a5c4800530"
Sep 17 19:01:04.721434 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.721413    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"]
Sep 17 19:01:04.725762 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:04.725738    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sgqgq"]
Sep 17 19:01:05.239572 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:05.239540    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" path="/var/lib/kubelet/pods/a9cf6209-1754-4007-ad76-44e65c5af89a/volumes"
Sep 17 19:01:14.631412 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:14.631372    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:01:23.439916 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.439865    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"]
Sep 17 19:01:23.440298 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.440257    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kube-rbac-proxy"
Sep 17 19:01:23.440298 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.440269    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kube-rbac-proxy"
Sep 17 19:01:23.440298 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.440277    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kserve-container"
Sep 17 19:01:23.440298 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.440282    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kserve-container"
Sep 17 19:01:23.440298 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.440290    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="storage-initializer"
Sep 17 19:01:23.440298 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.440296    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="storage-initializer"
Sep 17 19:01:23.440491 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.440358    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kserve-container"
Sep 17 19:01:23.440491 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.440365    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9cf6209-1754-4007-ad76-44e65c5af89a" containerName="kube-rbac-proxy"
Sep 17 19:01:23.443847 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.443825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.446493 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.446468    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 19:01:23.446602 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.446533    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 19:01:23.454014 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.453991    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"]
Sep 17 19:01:23.544118 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.544089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d544d572-b0f6-4b0d-b775-06fb5575fb31-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.544261 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.544131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.544261 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.544176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.544261 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.544206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnlqv\" (UniqueName: \"kubernetes.io/projected/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-api-access-rnlqv\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.544261 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.544229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d544d572-b0f6-4b0d-b775-06fb5575fb31-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.644757 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.644720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d544d572-b0f6-4b0d-b775-06fb5575fb31-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.644954 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.644762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.644954 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.644801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.644954 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.644832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnlqv\" (UniqueName: \"kubernetes.io/projected/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-api-access-rnlqv\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.644954 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.644856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d544d572-b0f6-4b0d-b775-06fb5575fb31-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.645358 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.645336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d544d572-b0f6-4b0d-b775-06fb5575fb31-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.645557 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.645537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.645601 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.645547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.647345 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.647322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d544d572-b0f6-4b0d-b775-06fb5575fb31-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.652253 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.652231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnlqv\" (UniqueName: \"kubernetes.io/projected/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-api-access-rnlqv\") pod \"isvc-triton-predictor-5d7db58676-j6s87\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:23.755044 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:23.754946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:01:24.028519 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:24.028490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"]
Sep 17 19:01:24.030756 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:01:24.030732    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd544d572_b0f6_4b0d_b775_06fb5575fb31.slice/crio-a3c1dc0885be7093d2930159e59bfdd83a0f212084a9d93666803267bda85705 WatchSource:0}: Error finding container a3c1dc0885be7093d2930159e59bfdd83a0f212084a9d93666803267bda85705: Status 404 returned error can't find the container with id a3c1dc0885be7093d2930159e59bfdd83a0f212084a9d93666803267bda85705
Sep 17 19:01:24.630581 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:24.630544    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:01:24.800061 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:24.800021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" event={"ID":"d544d572-b0f6-4b0d-b775-06fb5575fb31","Type":"ContainerStarted","Data":"2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3"}
Sep 17 19:01:24.800238 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:24.800080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" event={"ID":"d544d572-b0f6-4b0d-b775-06fb5575fb31","Type":"ContainerStarted","Data":"a3c1dc0885be7093d2930159e59bfdd83a0f212084a9d93666803267bda85705"}
Sep 17 19:01:28.821704 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:28.821663    2577 generic.go:356] "Generic (PLEG): container finished" podID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerID="2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3" exitCode=0
Sep 17 19:01:28.822128 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:28.821735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" event={"ID":"d544d572-b0f6-4b0d-b775-06fb5575fb31","Type":"ContainerDied","Data":"2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3"}
Sep 17 19:01:34.634326 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:34.634275    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:01:44.631615 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:44.631488    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:01:54.631127 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:01:54.630718    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:02:04.630609 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:04.630563    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:02:07.237516 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:07.237438    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:02:17.239622 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:17.239539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:02:22.129981 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:22.129941    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"]
Sep 17 19:02:22.130485 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:22.130277    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" containerID="cri-o://b3123d58c03c6400ec4fc2c704c49e5f23ad54482bbe3b7ad24b9cb81cf5bf11" gracePeriod=30
Sep 17 19:02:22.130485 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:22.130432    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kube-rbac-proxy" containerID="cri-o://ff505210b26795de988452298b6ad84d2606cb03d89f9be50d2d3eb0cc66ac74" gracePeriod=30
Sep 17 19:02:24.625608 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:24.625558    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 19:02:27.240216 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.240121    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:02:27.433183 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.433144    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"]
Sep 17 19:02:27.451843 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.451818    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"]
Sep 17 19:02:27.452032 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.451942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.455235 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.455188    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 19:02:27.455563 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.455534    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 19:02:27.592377 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.592307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.592377 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.592369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xht\" (UniqueName: \"kubernetes.io/projected/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-api-access-g9xht\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.592587 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.592391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a6217dc-b36e-4778-b981-76dcdecfc191-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.592587 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.592460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a6217dc-b36e-4778-b981-76dcdecfc191-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.592587 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.592515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.693926 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.693876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xht\" (UniqueName: \"kubernetes.io/projected/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-api-access-g9xht\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.694092 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.693933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a6217dc-b36e-4778-b981-76dcdecfc191-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.694092 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.693994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a6217dc-b36e-4778-b981-76dcdecfc191-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.694092 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.694064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.694281 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.694109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.694445 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.694421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a6217dc-b36e-4778-b981-76dcdecfc191-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.694923 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.694874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.694923 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.694890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.696789 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.696752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a6217dc-b36e-4778-b981-76dcdecfc191-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.702705 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.702683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xht\" (UniqueName: \"kubernetes.io/projected/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-api-access-g9xht\") pod \"isvc-canary-stop-predictor-7968cbc88d-h778t\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:27.764788 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:27.764764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:02:29.625719 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:29.625679    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 19:02:34.625005 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:34.624943    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 19:02:34.625507 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:34.625127    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:02:37.238498 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:37.238453    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:02:39.625442 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:39.625396    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 19:02:40.271225 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.271191    2577 generic.go:356] "Generic (PLEG): container finished" podID="396329aa-884b-4a4f-8095-34081d02c86d" containerID="ff505210b26795de988452298b6ad84d2606cb03d89f9be50d2d3eb0cc66ac74" exitCode=2
Sep 17 19:02:40.271225 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.271225    2577 generic.go:356] "Generic (PLEG): container finished" podID="396329aa-884b-4a4f-8095-34081d02c86d" containerID="b3123d58c03c6400ec4fc2c704c49e5f23ad54482bbe3b7ad24b9cb81cf5bf11" exitCode=0
Sep 17 19:02:40.271364 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.271254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" event={"ID":"396329aa-884b-4a4f-8095-34081d02c86d","Type":"ContainerDied","Data":"ff505210b26795de988452298b6ad84d2606cb03d89f9be50d2d3eb0cc66ac74"}
Sep 17 19:02:40.271364 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.271319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" event={"ID":"396329aa-884b-4a4f-8095-34081d02c86d","Type":"ContainerDied","Data":"b3123d58c03c6400ec4fc2c704c49e5f23ad54482bbe3b7ad24b9cb81cf5bf11"}
Sep 17 19:02:40.418573 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.418512    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:02:40.513179 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.513142    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqf8b\" (UniqueName: \"kubernetes.io/projected/396329aa-884b-4a4f-8095-34081d02c86d-kube-api-access-pqf8b\") pod \"396329aa-884b-4a4f-8095-34081d02c86d\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") "
Sep 17 19:02:40.513179 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.513188    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396329aa-884b-4a4f-8095-34081d02c86d-kserve-provision-location\") pod \"396329aa-884b-4a4f-8095-34081d02c86d\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") "
Sep 17 19:02:40.513416 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.513234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-cabundle-cert\") pod \"396329aa-884b-4a4f-8095-34081d02c86d\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") "
Sep 17 19:02:40.513478 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.513404    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-kube-rbac-proxy-sar-config\") pod \"396329aa-884b-4a4f-8095-34081d02c86d\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") "
Sep 17 19:02:40.513478 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.513447    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396329aa-884b-4a4f-8095-34081d02c86d-proxy-tls\") pod \"396329aa-884b-4a4f-8095-34081d02c86d\" (UID: \"396329aa-884b-4a4f-8095-34081d02c86d\") "
Sep 17 19:02:40.513584 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.513500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "396329aa-884b-4a4f-8095-34081d02c86d" (UID: "396329aa-884b-4a4f-8095-34081d02c86d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:40.513584 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.513550    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396329aa-884b-4a4f-8095-34081d02c86d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "396329aa-884b-4a4f-8095-34081d02c86d" (UID: "396329aa-884b-4a4f-8095-34081d02c86d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:40.513809 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.513790    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396329aa-884b-4a4f-8095-34081d02c86d-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:02:40.513809 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.513803    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "396329aa-884b-4a4f-8095-34081d02c86d" (UID: "396329aa-884b-4a4f-8095-34081d02c86d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:40.513959 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.513813    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:02:40.515548 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.515522    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396329aa-884b-4a4f-8095-34081d02c86d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "396329aa-884b-4a4f-8095-34081d02c86d" (UID: "396329aa-884b-4a4f-8095-34081d02c86d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:40.515706 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.515684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396329aa-884b-4a4f-8095-34081d02c86d-kube-api-access-pqf8b" (OuterVolumeSpecName: "kube-api-access-pqf8b") pod "396329aa-884b-4a4f-8095-34081d02c86d" (UID: "396329aa-884b-4a4f-8095-34081d02c86d"). InnerVolumeSpecName "kube-api-access-pqf8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:40.615092 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.615061    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396329aa-884b-4a4f-8095-34081d02c86d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:02:40.615092 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.615093    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396329aa-884b-4a4f-8095-34081d02c86d-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:02:40.615289 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.615107    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqf8b\" (UniqueName: \"kubernetes.io/projected/396329aa-884b-4a4f-8095-34081d02c86d-kube-api-access-pqf8b\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:02:40.747547 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:40.747433    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"]
Sep 17 19:02:41.273667 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:02:41.273639    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6217dc_b36e_4778_b981_76dcdecfc191.slice/crio-231815062c3ec6c117ffec87969d1f78560cdd95c3d4979154d736744fb49833 WatchSource:0}: Error finding container 231815062c3ec6c117ffec87969d1f78560cdd95c3d4979154d736744fb49833: Status 404 returned error can't find the container with id 231815062c3ec6c117ffec87969d1f78560cdd95c3d4979154d736744fb49833
Sep 17 19:02:41.278179 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:41.278153    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"
Sep 17 19:02:41.278179 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:41.278160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc" event={"ID":"396329aa-884b-4a4f-8095-34081d02c86d","Type":"ContainerDied","Data":"8be0538e29a11598d2be17345f3c1353701f58bb6d34bd8c2a74e2a169e528c1"}
Sep 17 19:02:41.278332 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:41.278254    2577 scope.go:117] "RemoveContainer" containerID="ff505210b26795de988452298b6ad84d2606cb03d89f9be50d2d3eb0cc66ac74"
Sep 17 19:02:41.298609 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:41.298586    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"]
Sep 17 19:02:41.303734 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:41.303712    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-kwwrc"]
Sep 17 19:02:42.285105 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:42.285073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" event={"ID":"0a6217dc-b36e-4778-b981-76dcdecfc191","Type":"ContainerStarted","Data":"231815062c3ec6c117ffec87969d1f78560cdd95c3d4979154d736744fb49833"}
Sep 17 19:02:42.395658 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:42.395637    2577 scope.go:117] "RemoveContainer" containerID="b3123d58c03c6400ec4fc2c704c49e5f23ad54482bbe3b7ad24b9cb81cf5bf11"
Sep 17 19:02:43.239963 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:43.239934    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="396329aa-884b-4a4f-8095-34081d02c86d" path="/var/lib/kubelet/pods/396329aa-884b-4a4f-8095-34081d02c86d/volumes"
Sep 17 19:02:45.969305 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:45.969006    2577 scope.go:117] "RemoveContainer" containerID="ccac3ddb650eb332857cdb9b07590abd4c978777671c5fda075d7bd8d1ce6ca3"
Sep 17 19:02:48.322199 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:48.322154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" event={"ID":"0a6217dc-b36e-4778-b981-76dcdecfc191","Type":"ContainerStarted","Data":"d202b292be039c68926d910a23f4f37e9d6dbcba4499a2433239324000140dc0"}
Sep 17 19:02:52.350152 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:52.350037    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerID="d202b292be039c68926d910a23f4f37e9d6dbcba4499a2433239324000140dc0" exitCode=0
Sep 17 19:02:52.350152 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:02:52.350108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" event={"ID":"0a6217dc-b36e-4778-b981-76dcdecfc191","Type":"ContainerDied","Data":"d202b292be039c68926d910a23f4f37e9d6dbcba4499a2433239324000140dc0"}
Sep 17 19:03:20.501924 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:20.501833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" event={"ID":"0a6217dc-b36e-4778-b981-76dcdecfc191","Type":"ContainerStarted","Data":"b876f60743dec399efe5d11563091fc12c1a8aadcd64abd2c61f311d84a4737e"}
Sep 17 19:03:20.502277 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:20.501931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" event={"ID":"0a6217dc-b36e-4778-b981-76dcdecfc191","Type":"ContainerStarted","Data":"7b69832e058140f7a8194bbb032f7da240990d28f3b47422325413520c647847"}
Sep 17 19:03:21.509770 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:21.509665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" event={"ID":"d544d572-b0f6-4b0d-b775-06fb5575fb31","Type":"ContainerStarted","Data":"8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835"}
Sep 17 19:03:21.510287 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:21.509932    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:03:21.510287 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:21.509988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:03:21.510287 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:21.510016    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:03:21.510287 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:21.510045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" event={"ID":"d544d572-b0f6-4b0d-b775-06fb5575fb31","Type":"ContainerStarted","Data":"9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d"}
Sep 17 19:03:21.511397 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:21.511370    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:03:21.531858 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:21.531810    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" podStartSLOduration=2.094490945 podStartE2EDuration="1m58.531793746s" totalImagesPullingTime="1m52.417794624s" totalInitContainerRuntime="4.019508177s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:21.530074024 +0000 UTC m=+2036.576476936" watchObservedRunningTime="2026-09-17 19:03:21.531793746 +0000 UTC m=+2036.578196647"
Sep 17 19:03:22.514926 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:22.514862    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:03:22.515397 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:22.515014    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:03:22.516485 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:22.516458    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:03:23.519565 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:23.519520    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:03:27.519873 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:27.519843    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:03:27.520383 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:27.520358    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:03:27.542412 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:27.542370    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podStartSLOduration=55.994719963 podStartE2EDuration="1m0.542357581s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.547637618s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:21.556984552 +0000 UTC m=+2036.603387481" watchObservedRunningTime="2026-09-17 19:03:27.542357581 +0000 UTC m=+2042.588760509"
Sep 17 19:03:28.525144 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:28.525112    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:03:28.525829 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:28.525801    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:03:37.521359 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:37.521328    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:03:38.526140 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:38.526056    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:03:45.583650 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:45.583604    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"]
Sep 17 19:03:45.584292 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:45.583921    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kserve-container" containerID="cri-o://9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d" gracePeriod=30
Sep 17 19:03:45.584292 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:45.584017    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kube-rbac-proxy" containerID="cri-o://8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835" gracePeriod=30
Sep 17 19:03:46.632128 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:46.632093    2577 generic.go:356] "Generic (PLEG): container finished" podID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerID="8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835" exitCode=2
Sep 17 19:03:46.632535 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:46.632175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" event={"ID":"d544d572-b0f6-4b0d-b775-06fb5575fb31","Type":"ContainerDied","Data":"8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835"}
Sep 17 19:03:47.515445 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:47.515402    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 17 19:03:47.520674 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:47.520649    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:03:48.525996 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.525957    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:03:48.559593 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.559559    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:03:48.644244 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.644217    2577 generic.go:356] "Generic (PLEG): container finished" podID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerID="9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d" exitCode=0
Sep 17 19:03:48.644410 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.644271    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"
Sep 17 19:03:48.644410 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.644294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" event={"ID":"d544d572-b0f6-4b0d-b775-06fb5575fb31","Type":"ContainerDied","Data":"9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d"}
Sep 17 19:03:48.644410 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.644363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87" event={"ID":"d544d572-b0f6-4b0d-b775-06fb5575fb31","Type":"ContainerDied","Data":"a3c1dc0885be7093d2930159e59bfdd83a0f212084a9d93666803267bda85705"}
Sep 17 19:03:48.644410 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.644403    2577 scope.go:117] "RemoveContainer" containerID="8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835"
Sep 17 19:03:48.650184 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.650162    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-rbac-proxy-sar-config\") pod \"d544d572-b0f6-4b0d-b775-06fb5575fb31\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") "
Sep 17 19:03:48.650304 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.650212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-cabundle-cert\") pod \"d544d572-b0f6-4b0d-b775-06fb5575fb31\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") "
Sep 17 19:03:48.650304 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.650270    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d544d572-b0f6-4b0d-b775-06fb5575fb31-kserve-provision-location\") pod \"d544d572-b0f6-4b0d-b775-06fb5575fb31\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") "
Sep 17 19:03:48.650395 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.650341    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d544d572-b0f6-4b0d-b775-06fb5575fb31-proxy-tls\") pod \"d544d572-b0f6-4b0d-b775-06fb5575fb31\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") "
Sep 17 19:03:48.650436 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.650406    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnlqv\" (UniqueName: \"kubernetes.io/projected/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-api-access-rnlqv\") pod \"d544d572-b0f6-4b0d-b775-06fb5575fb31\" (UID: \"d544d572-b0f6-4b0d-b775-06fb5575fb31\") "
Sep 17 19:03:48.650647 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.650617    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d544d572-b0f6-4b0d-b775-06fb5575fb31" (UID: "d544d572-b0f6-4b0d-b775-06fb5575fb31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:03:48.650647 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.650627    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d544d572-b0f6-4b0d-b775-06fb5575fb31" (UID: "d544d572-b0f6-4b0d-b775-06fb5575fb31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:03:48.650808 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.650781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d544d572-b0f6-4b0d-b775-06fb5575fb31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d544d572-b0f6-4b0d-b775-06fb5575fb31" (UID: "d544d572-b0f6-4b0d-b775-06fb5575fb31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:48.652413 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.652380    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-api-access-rnlqv" (OuterVolumeSpecName: "kube-api-access-rnlqv") pod "d544d572-b0f6-4b0d-b775-06fb5575fb31" (UID: "d544d572-b0f6-4b0d-b775-06fb5575fb31"). InnerVolumeSpecName "kube-api-access-rnlqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:03:48.652701 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.652681    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d544d572-b0f6-4b0d-b775-06fb5575fb31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d544d572-b0f6-4b0d-b775-06fb5575fb31" (UID: "d544d572-b0f6-4b0d-b775-06fb5575fb31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:48.653608 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.653593    2577 scope.go:117] "RemoveContainer" containerID="9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d"
Sep 17 19:03:48.667763 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.667745    2577 scope.go:117] "RemoveContainer" containerID="2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3"
Sep 17 19:03:48.675442 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.675426    2577 scope.go:117] "RemoveContainer" containerID="8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835"
Sep 17 19:03:48.675685 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:03:48.675666    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835\": container with ID starting with 8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835 not found: ID does not exist" containerID="8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835"
Sep 17 19:03:48.675732 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.675693    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835"} err="failed to get container status \"8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835\": rpc error: code = NotFound desc = could not find container \"8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835\": container with ID starting with 8c0027f8aba687d502c99124bdae015d69e125c0210679d84852197d9fa67835 not found: ID does not exist"
Sep 17 19:03:48.675732 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.675712    2577 scope.go:117] "RemoveContainer" containerID="9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d"
Sep 17 19:03:48.675968 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:03:48.675952    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d\": container with ID starting with 9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d not found: ID does not exist" containerID="9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d"
Sep 17 19:03:48.676025 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.675970    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d"} err="failed to get container status \"9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d\": rpc error: code = NotFound desc = could not find container \"9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d\": container with ID starting with 9b7fd58d9c1522cda2500ed319ac318fc9970c9121521a1193ae05fc917b4e4d not found: ID does not exist"
Sep 17 19:03:48.676025 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.675984    2577 scope.go:117] "RemoveContainer" containerID="2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3"
Sep 17 19:03:48.676213 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:03:48.676194    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3\": container with ID starting with 2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3 not found: ID does not exist" containerID="2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3"
Sep 17 19:03:48.676254 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.676219    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3"} err="failed to get container status \"2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3\": rpc error: code = NotFound desc = could not find container \"2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3\": container with ID starting with 2f5e7993f5a634a1c063fc57d0e295743a5d5b75fbc0324b7495245e691ba4f3 not found: ID does not exist"
Sep 17 19:03:48.751400 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.751368    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d544d572-b0f6-4b0d-b775-06fb5575fb31-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:03:48.751400 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.751394    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnlqv\" (UniqueName: \"kubernetes.io/projected/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-api-access-rnlqv\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:03:48.751400 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.751404    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:03:48.751636 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.751415    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d544d572-b0f6-4b0d-b775-06fb5575fb31-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:03:48.751636 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.751425    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d544d572-b0f6-4b0d-b775-06fb5575fb31-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:03:48.967719 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.967687    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"]
Sep 17 19:03:48.971584 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:48.971556    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-j6s87"]
Sep 17 19:03:49.238823 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:49.238746    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" path="/var/lib/kubelet/pods/d544d572-b0f6-4b0d-b775-06fb5575fb31/volumes"
Sep 17 19:03:50.765314 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765279    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"]
Sep 17 19:03:50.765707 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765649    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="storage-initializer"
Sep 17 19:03:50.765707 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765660    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="storage-initializer"
Sep 17 19:03:50.765707 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765670    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kube-rbac-proxy"
Sep 17 19:03:50.765707 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765675    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kube-rbac-proxy"
Sep 17 19:03:50.765707 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765686    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="storage-initializer"
Sep 17 19:03:50.765707 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765692    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="storage-initializer"
Sep 17 19:03:50.765707 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765698    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container"
Sep 17 19:03:50.765707 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765704    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container"
Sep 17 19:03:50.765982 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765713    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kube-rbac-proxy"
Sep 17 19:03:50.765982 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765720    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kube-rbac-proxy"
Sep 17 19:03:50.765982 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765737    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kserve-container"
Sep 17 19:03:50.765982 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765742    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kserve-container"
Sep 17 19:03:50.765982 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765811    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kserve-container"
Sep 17 19:03:50.765982 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765818    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kube-rbac-proxy"
Sep 17 19:03:50.765982 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765826    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="396329aa-884b-4a4f-8095-34081d02c86d" containerName="kserve-container"
Sep 17 19:03:50.765982 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.765832    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d544d572-b0f6-4b0d-b775-06fb5575fb31" containerName="kube-rbac-proxy"
Sep 17 19:03:50.782917 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.782872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.786096 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.786048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 19:03:50.786315 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.786290    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 19:03:50.787779 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.787751    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"]
Sep 17 19:03:50.870959 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.870897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.871132 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.871020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxvq9\" (UniqueName: \"kubernetes.io/projected/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-api-access-lxvq9\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.871132 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.871070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.871132 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.871090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.871252 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.871138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddded7fc-e42d-4d31-aac6-b180dddfd90f-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.971610 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.971572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxvq9\" (UniqueName: \"kubernetes.io/projected/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-api-access-lxvq9\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.971782 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.971626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.971782 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.971654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.971782 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.971688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddded7fc-e42d-4d31-aac6-b180dddfd90f-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.971782 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.971756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.972182 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.972157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.972303 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.972284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.972389 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.972365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.974206 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.974183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddded7fc-e42d-4d31-aac6-b180dddfd90f-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:50.979705 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:50.979685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxvq9\" (UniqueName: \"kubernetes.io/projected/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-api-access-lxvq9\") pod \"isvc-xgboost-predictor-7566d65bf7-knzdn\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:51.094201 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:51.094117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:03:51.148941 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:51.148881    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"]
Sep 17 19:03:51.149234 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:51.149196    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container" containerID="cri-o://7b69832e058140f7a8194bbb032f7da240990d28f3b47422325413520c647847" gracePeriod=30
Sep 17 19:03:51.149535 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:51.149456    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:03:51.149535 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:51.149256    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kube-rbac-proxy" containerID="cri-o://b876f60743dec399efe5d11563091fc12c1a8aadcd64abd2c61f311d84a4737e" gracePeriod=30
Sep 17 19:03:51.388106 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:51.388080    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"]
Sep 17 19:03:51.390847 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:03:51.390821    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddded7fc_e42d_4d31_aac6_b180dddfd90f.slice/crio-f9450727593b367a4f1b3734eed71c848a3e2873de7db24a576a7ea6e5794a50 WatchSource:0}: Error finding container f9450727593b367a4f1b3734eed71c848a3e2873de7db24a576a7ea6e5794a50: Status 404 returned error can't find the container with id f9450727593b367a4f1b3734eed71c848a3e2873de7db24a576a7ea6e5794a50
Sep 17 19:03:51.666263 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:51.666176    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerID="b876f60743dec399efe5d11563091fc12c1a8aadcd64abd2c61f311d84a4737e" exitCode=2
Sep 17 19:03:51.666471 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:51.666256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" event={"ID":"0a6217dc-b36e-4778-b981-76dcdecfc191","Type":"ContainerDied","Data":"b876f60743dec399efe5d11563091fc12c1a8aadcd64abd2c61f311d84a4737e"}
Sep 17 19:03:51.667787 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:51.667758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" event={"ID":"ddded7fc-e42d-4d31-aac6-b180dddfd90f","Type":"ContainerStarted","Data":"e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8"}
Sep 17 19:03:51.667929 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:51.667802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" event={"ID":"ddded7fc-e42d-4d31-aac6-b180dddfd90f","Type":"ContainerStarted","Data":"f9450727593b367a4f1b3734eed71c848a3e2873de7db24a576a7ea6e5794a50"}
Sep 17 19:03:53.520649 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:53.520601    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:03:56.693321 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:56.693280    2577 generic.go:356] "Generic (PLEG): container finished" podID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerID="e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8" exitCode=0
Sep 17 19:03:56.693790 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:56.693341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" event={"ID":"ddded7fc-e42d-4d31-aac6-b180dddfd90f","Type":"ContainerDied","Data":"e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8"}
Sep 17 19:03:58.520314 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:03:58.520266    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:04:01.150288 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:01.150238    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:04:03.520810 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:03.520763    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:04:03.521392 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:03.520942    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:04:08.520610 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:08.520566    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:04:11.149892 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:11.149842    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:04:13.520095 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:13.520052    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:04:18.520067 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:18.520023    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:04:19.820572 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:19.820485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" event={"ID":"ddded7fc-e42d-4d31-aac6-b180dddfd90f","Type":"ContainerStarted","Data":"855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609"}
Sep 17 19:04:19.820572 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:19.820552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" event={"ID":"ddded7fc-e42d-4d31-aac6-b180dddfd90f","Type":"ContainerStarted","Data":"58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97"}
Sep 17 19:04:19.820965 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:19.820796    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:04:19.842972 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:19.842925    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podStartSLOduration=2.7536876120000002 podStartE2EDuration="29.842897809s" totalImagesPullingTime="22.823652279s" totalInitContainerRuntime="4.265557918s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:19.840634872 +0000 UTC m=+2094.887037788" watchObservedRunningTime="2026-09-17 19:04:19.842897809 +0000 UTC m=+2094.889300710"
Sep 17 19:04:20.825623 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:20.825593    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:04:20.826856 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:20.826827    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 19:04:21.149767 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.149684    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:04:21.832552 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.832526    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerID="7b69832e058140f7a8194bbb032f7da240990d28f3b47422325413520c647847" exitCode=137
Sep 17 19:04:21.832889 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.832599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" event={"ID":"0a6217dc-b36e-4778-b981-76dcdecfc191","Type":"ContainerDied","Data":"7b69832e058140f7a8194bbb032f7da240990d28f3b47422325413520c647847"}
Sep 17 19:04:21.832889 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.832671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t" event={"ID":"0a6217dc-b36e-4778-b981-76dcdecfc191","Type":"ContainerDied","Data":"231815062c3ec6c117ffec87969d1f78560cdd95c3d4979154d736744fb49833"}
Sep 17 19:04:21.832889 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.832706    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="231815062c3ec6c117ffec87969d1f78560cdd95c3d4979154d736744fb49833"
Sep 17 19:04:21.833153 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.832891    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 19:04:21.840155 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.840139    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:04:21.962750 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.962681    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-rbac-proxy-sar-config\") pod \"0a6217dc-b36e-4778-b981-76dcdecfc191\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") "
Sep 17 19:04:21.962750 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.962738    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a6217dc-b36e-4778-b981-76dcdecfc191-kserve-provision-location\") pod \"0a6217dc-b36e-4778-b981-76dcdecfc191\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") "
Sep 17 19:04:21.962987 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.962795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-cabundle-cert\") pod \"0a6217dc-b36e-4778-b981-76dcdecfc191\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") "
Sep 17 19:04:21.962987 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.962820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a6217dc-b36e-4778-b981-76dcdecfc191-proxy-tls\") pod \"0a6217dc-b36e-4778-b981-76dcdecfc191\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") "
Sep 17 19:04:21.962987 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.962844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9xht\" (UniqueName: \"kubernetes.io/projected/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-api-access-g9xht\") pod \"0a6217dc-b36e-4778-b981-76dcdecfc191\" (UID: \"0a6217dc-b36e-4778-b981-76dcdecfc191\") "
Sep 17 19:04:21.963143 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.963080    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6217dc-b36e-4778-b981-76dcdecfc191-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a6217dc-b36e-4778-b981-76dcdecfc191" (UID: "0a6217dc-b36e-4778-b981-76dcdecfc191"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:21.963202 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.963020    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0a6217dc-b36e-4778-b981-76dcdecfc191" (UID: "0a6217dc-b36e-4778-b981-76dcdecfc191"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:21.963202 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.963167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a6217dc-b36e-4778-b981-76dcdecfc191" (UID: "0a6217dc-b36e-4778-b981-76dcdecfc191"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:21.963336 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.963309    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:04:21.963387 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.963336    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a6217dc-b36e-4778-b981-76dcdecfc191-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:04:21.963387 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.963350    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a6217dc-b36e-4778-b981-76dcdecfc191-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:04:21.965026 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.965005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-api-access-g9xht" (OuterVolumeSpecName: "kube-api-access-g9xht") pod "0a6217dc-b36e-4778-b981-76dcdecfc191" (UID: "0a6217dc-b36e-4778-b981-76dcdecfc191"). InnerVolumeSpecName "kube-api-access-g9xht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:21.965121 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:21.965031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a6217dc-b36e-4778-b981-76dcdecfc191-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a6217dc-b36e-4778-b981-76dcdecfc191" (UID: "0a6217dc-b36e-4778-b981-76dcdecfc191"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:22.063966 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:22.063937    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a6217dc-b36e-4778-b981-76dcdecfc191-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:04:22.063966 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:22.063959    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xht\" (UniqueName: \"kubernetes.io/projected/0a6217dc-b36e-4778-b981-76dcdecfc191-kube-api-access-g9xht\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:04:22.837576 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:22.837544    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"
Sep 17 19:04:22.861518 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:22.861480    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"]
Sep 17 19:04:22.868114 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:22.868083    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h778t"]
Sep 17 19:04:23.242666 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:23.242629    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" path="/var/lib/kubelet/pods/0a6217dc-b36e-4778-b981-76dcdecfc191/volumes"
Sep 17 19:04:26.837720 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:26.837685    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:04:26.838437 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:26.838405    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 19:04:36.839057 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:36.839017    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 19:04:46.838785 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:46.838746    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 19:04:56.838822 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:04:56.838786    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 19:05:06.839171 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:06.839124    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 19:05:16.838673 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:16.838587    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 19:05:26.839714 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:26.839686    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:05:31.202108 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:31.202075    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"]
Sep 17 19:05:31.202528 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:31.202324    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container" containerID="cri-o://58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97" gracePeriod=30
Sep 17 19:05:31.202528 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:31.202375    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kube-rbac-proxy" containerID="cri-o://855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609" gracePeriod=30
Sep 17 19:05:31.833918 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:31.833869    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 17 19:05:32.170497 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:32.170416    2577 generic.go:356] "Generic (PLEG): container finished" podID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerID="855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609" exitCode=2
Sep 17 19:05:32.170497 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:32.170458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" event={"ID":"ddded7fc-e42d-4d31-aac6-b180dddfd90f","Type":"ContainerDied","Data":"855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609"}
Sep 17 19:05:35.043509 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.043480    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:05:35.085944 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.085888    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-rbac-proxy-sar-config\") pod \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") "
Sep 17 19:05:35.086083 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.085967    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kserve-provision-location\") pod \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") "
Sep 17 19:05:35.086083 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.086027    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-cabundle-cert\") pod \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") "
Sep 17 19:05:35.086240 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.086215    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ddded7fc-e42d-4d31-aac6-b180dddfd90f" (UID: "ddded7fc-e42d-4d31-aac6-b180dddfd90f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:05:35.086310 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.086300    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:05:35.086359 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.086308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ddded7fc-e42d-4d31-aac6-b180dddfd90f" (UID: "ddded7fc-e42d-4d31-aac6-b180dddfd90f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:35.086396 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.086371    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ddded7fc-e42d-4d31-aac6-b180dddfd90f" (UID: "ddded7fc-e42d-4d31-aac6-b180dddfd90f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:05:35.186556 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.186530    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxvq9\" (UniqueName: \"kubernetes.io/projected/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-api-access-lxvq9\") pod \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") "
Sep 17 19:05:35.186706 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.186659    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddded7fc-e42d-4d31-aac6-b180dddfd90f-proxy-tls\") pod \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\" (UID: \"ddded7fc-e42d-4d31-aac6-b180dddfd90f\") "
Sep 17 19:05:35.186856 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.186841    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:05:35.186856 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.186860    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddded7fc-e42d-4d31-aac6-b180dddfd90f-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:05:35.188218 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.188194    2577 generic.go:356] "Generic (PLEG): container finished" podID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerID="58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97" exitCode=0
Sep 17 19:05:35.188330 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.188240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" event={"ID":"ddded7fc-e42d-4d31-aac6-b180dddfd90f","Type":"ContainerDied","Data":"58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97"}
Sep 17 19:05:35.188330 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.188262    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"
Sep 17 19:05:35.188330 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.188305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn" event={"ID":"ddded7fc-e42d-4d31-aac6-b180dddfd90f","Type":"ContainerDied","Data":"f9450727593b367a4f1b3734eed71c848a3e2873de7db24a576a7ea6e5794a50"}
Sep 17 19:05:35.188524 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.188342    2577 scope.go:117] "RemoveContainer" containerID="855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609"
Sep 17 19:05:35.188844 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.188816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-api-access-lxvq9" (OuterVolumeSpecName: "kube-api-access-lxvq9") pod "ddded7fc-e42d-4d31-aac6-b180dddfd90f" (UID: "ddded7fc-e42d-4d31-aac6-b180dddfd90f"). InnerVolumeSpecName "kube-api-access-lxvq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:05:35.188963 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.188946    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddded7fc-e42d-4d31-aac6-b180dddfd90f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ddded7fc-e42d-4d31-aac6-b180dddfd90f" (UID: "ddded7fc-e42d-4d31-aac6-b180dddfd90f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:05:35.204161 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.204130    2577 scope.go:117] "RemoveContainer" containerID="58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97"
Sep 17 19:05:35.211201 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.211187    2577 scope.go:117] "RemoveContainer" containerID="e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8"
Sep 17 19:05:35.218212 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.218193    2577 scope.go:117] "RemoveContainer" containerID="855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609"
Sep 17 19:05:35.218468 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:05:35.218449    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609\": container with ID starting with 855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609 not found: ID does not exist" containerID="855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609"
Sep 17 19:05:35.218512 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.218479    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609"} err="failed to get container status \"855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609\": rpc error: code = NotFound desc = could not find container \"855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609\": container with ID starting with 855d63d83ddc7cc2ac331d486046c9bb1da2fac9033524f51d28e04eb455d609 not found: ID does not exist"
Sep 17 19:05:35.218512 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.218499    2577 scope.go:117] "RemoveContainer" containerID="58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97"
Sep 17 19:05:35.218742 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:05:35.218724    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97\": container with ID starting with 58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97 not found: ID does not exist" containerID="58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97"
Sep 17 19:05:35.218784 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.218747    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97"} err="failed to get container status \"58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97\": rpc error: code = NotFound desc = could not find container \"58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97\": container with ID starting with 58e8f0f654540848c4cebf3e0b56bc1dcd5e175b22eb464399bcd2db06b62d97 not found: ID does not exist"
Sep 17 19:05:35.218784 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.218763    2577 scope.go:117] "RemoveContainer" containerID="e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8"
Sep 17 19:05:35.219034 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:05:35.219016    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8\": container with ID starting with e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8 not found: ID does not exist" containerID="e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8"
Sep 17 19:05:35.219087 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.219039    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8"} err="failed to get container status \"e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8\": rpc error: code = NotFound desc = could not find container \"e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8\": container with ID starting with e777e8b0f115b5836de4bfd8d5b2c7b1a6627dce8dc2751ff41a261e88b9fbc8 not found: ID does not exist"
Sep 17 19:05:35.287603 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.287578    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddded7fc-e42d-4d31-aac6-b180dddfd90f-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:05:35.287603 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.287600    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxvq9\" (UniqueName: \"kubernetes.io/projected/ddded7fc-e42d-4d31-aac6-b180dddfd90f-kube-api-access-lxvq9\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:05:35.508604 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.508525    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"]
Sep 17 19:05:35.513552 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:35.513528    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-knzdn"]
Sep 17 19:05:37.239010 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:05:37.238975    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" path="/var/lib/kubelet/pods/ddded7fc-e42d-4d31-aac6-b180dddfd90f/volumes"
Sep 17 19:09:31.983764 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:09:31.983728    2577 scope.go:117] "RemoveContainer" containerID="b876f60743dec399efe5d11563091fc12c1a8aadcd64abd2c61f311d84a4737e"
Sep 17 19:09:31.991929 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:09:31.991879    2577 scope.go:117] "RemoveContainer" containerID="d202b292be039c68926d910a23f4f37e9d6dbcba4499a2433239324000140dc0"
Sep 17 19:09:31.999305 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:09:31.999287    2577 scope.go:117] "RemoveContainer" containerID="7b69832e058140f7a8194bbb032f7da240990d28f3b47422325413520c647847"
Sep 17 19:10:17.998925 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.998868    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"]
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999243    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="storage-initializer"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999257    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="storage-initializer"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999268    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999273    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999285    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kube-rbac-proxy"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999291    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kube-rbac-proxy"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999302    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kube-rbac-proxy"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999308    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kube-rbac-proxy"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999320    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="storage-initializer"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999325    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="storage-initializer"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999333    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999338    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999389    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kserve-container"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999395    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a6217dc-b36e-4778-b981-76dcdecfc191" containerName="kube-rbac-proxy"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999404    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kserve-container"
Sep 17 19:10:17.999408 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:17.999413    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddded7fc-e42d-4d31-aac6-b180dddfd90f" containerName="kube-rbac-proxy"
Sep 17 19:10:18.002605 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.002578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.007878 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.007850    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:10:18.008064 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.007928    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 19:10:18.009365 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.009029    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:10:18.009365 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.009205    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:10:18.009365 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.009264    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 19:10:18.009365 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.009307    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 19:10:18.009608 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.009476    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:10:18.045593 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.045550    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"]
Sep 17 19:10:18.097896 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.097856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2a93444-064d-4e25-baae-06d59fa4007a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.097896 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.097897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a93444-064d-4e25-baae-06d59fa4007a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.098158 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.097968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.098158 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.098052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.098158 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.098068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjgbf\" (UniqueName: \"kubernetes.io/projected/f2a93444-064d-4e25-baae-06d59fa4007a-kube-api-access-fjgbf\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.199160 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.199120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.199160 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.199161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgbf\" (UniqueName: \"kubernetes.io/projected/f2a93444-064d-4e25-baae-06d59fa4007a-kube-api-access-fjgbf\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.199435 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.199298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2a93444-064d-4e25-baae-06d59fa4007a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.199435 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.199344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a93444-064d-4e25-baae-06d59fa4007a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.199435 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.199384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.199713 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.199692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a93444-064d-4e25-baae-06d59fa4007a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.199974 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.199955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.200074 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.199999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.201920 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.201887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2a93444-064d-4e25-baae-06d59fa4007a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.207776 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.207746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjgbf\" (UniqueName: \"kubernetes.io/projected/f2a93444-064d-4e25-baae-06d59fa4007a-kube-api-access-fjgbf\") pod \"isvc-xgboost-v2-predictor-fcc668bc-xjp52\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.312887 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.312790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:18.591376 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:18.591351    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"]
Sep 17 19:10:18.593600 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:10:18.593567    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a93444_064d_4e25_baae_06d59fa4007a.slice/crio-36ce4ee7c094cb82f22a2fd30e94789bf95b9a4f3ef2fd38f256032bd9279b45 WatchSource:0}: Error finding container 36ce4ee7c094cb82f22a2fd30e94789bf95b9a4f3ef2fd38f256032bd9279b45: Status 404 returned error can't find the container with id 36ce4ee7c094cb82f22a2fd30e94789bf95b9a4f3ef2fd38f256032bd9279b45
Sep 17 19:10:19.501038 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:19.501000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" event={"ID":"f2a93444-064d-4e25-baae-06d59fa4007a","Type":"ContainerStarted","Data":"cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad"}
Sep 17 19:10:19.501426 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:19.501068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" event={"ID":"f2a93444-064d-4e25-baae-06d59fa4007a","Type":"ContainerStarted","Data":"36ce4ee7c094cb82f22a2fd30e94789bf95b9a4f3ef2fd38f256032bd9279b45"}
Sep 17 19:10:22.518090 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:22.518052    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2a93444-064d-4e25-baae-06d59fa4007a" containerID="cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad" exitCode=0
Sep 17 19:10:22.518486 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:22.518126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" event={"ID":"f2a93444-064d-4e25-baae-06d59fa4007a","Type":"ContainerDied","Data":"cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad"}
Sep 17 19:10:23.525081 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:23.525047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" event={"ID":"f2a93444-064d-4e25-baae-06d59fa4007a","Type":"ContainerStarted","Data":"bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41"}
Sep 17 19:10:23.525574 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:23.525109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" event={"ID":"f2a93444-064d-4e25-baae-06d59fa4007a","Type":"ContainerStarted","Data":"6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7"}
Sep 17 19:10:23.525574 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:23.525375    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:23.525574 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:23.525516    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:23.526924 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:23.526874    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:10:23.553200 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:23.553144    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podStartSLOduration=2.731243341 podStartE2EDuration="6.553124291s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.82188095s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:23.552040349 +0000 UTC m=+2458.598443272" watchObservedRunningTime="2026-09-17 19:10:23.553124291 +0000 UTC m=+2458.599527194"
Sep 17 19:10:24.530151 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:24.530116    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:10:29.534788 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:29.534759    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:10:29.535303 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:29.535277    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:10:39.535390 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:39.535351    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:10:49.535719 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:49.535671    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:10:59.535327 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:10:59.535284    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:11:09.535600 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:09.535509    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:11:19.535442 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:19.535401    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:11:29.536675 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:29.536634    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:11:38.653643 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:38.653610    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"]
Sep 17 19:11:38.656020 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:38.653867    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container" containerID="cri-o://6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7" gracePeriod=30
Sep 17 19:11:38.656020 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:38.653974    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kube-rbac-proxy" containerID="cri-o://bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41" gracePeriod=30
Sep 17 19:11:38.897008 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:38.896973    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2a93444-064d-4e25-baae-06d59fa4007a" containerID="bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41" exitCode=2
Sep 17 19:11:38.897008 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:38.897010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" event={"ID":"f2a93444-064d-4e25-baae-06d59fa4007a","Type":"ContainerDied","Data":"bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41"}
Sep 17 19:11:39.531120 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:39.531085    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 17 19:11:39.535496 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:39.535470    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:11:42.795388 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.795361    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:11:42.855576 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.855496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-kube-rbac-proxy-sar-config\") pod \"f2a93444-064d-4e25-baae-06d59fa4007a\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") "
Sep 17 19:11:42.855576 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.855544    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-cabundle-cert\") pod \"f2a93444-064d-4e25-baae-06d59fa4007a\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") "
Sep 17 19:11:42.855796 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.855587    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a93444-064d-4e25-baae-06d59fa4007a-kserve-provision-location\") pod \"f2a93444-064d-4e25-baae-06d59fa4007a\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") "
Sep 17 19:11:42.855796 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.855631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjgbf\" (UniqueName: \"kubernetes.io/projected/f2a93444-064d-4e25-baae-06d59fa4007a-kube-api-access-fjgbf\") pod \"f2a93444-064d-4e25-baae-06d59fa4007a\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") "
Sep 17 19:11:42.855796 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.855652    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2a93444-064d-4e25-baae-06d59fa4007a-proxy-tls\") pod \"f2a93444-064d-4e25-baae-06d59fa4007a\" (UID: \"f2a93444-064d-4e25-baae-06d59fa4007a\") "
Sep 17 19:11:42.855994 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.855878    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2a93444-064d-4e25-baae-06d59fa4007a" (UID: "f2a93444-064d-4e25-baae-06d59fa4007a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:11:42.855994 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.855893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a93444-064d-4e25-baae-06d59fa4007a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2a93444-064d-4e25-baae-06d59fa4007a" (UID: "f2a93444-064d-4e25-baae-06d59fa4007a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:11:42.855994 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.855975    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2a93444-064d-4e25-baae-06d59fa4007a" (UID: "f2a93444-064d-4e25-baae-06d59fa4007a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:11:42.857724 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.857701    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a93444-064d-4e25-baae-06d59fa4007a-kube-api-access-fjgbf" (OuterVolumeSpecName: "kube-api-access-fjgbf") pod "f2a93444-064d-4e25-baae-06d59fa4007a" (UID: "f2a93444-064d-4e25-baae-06d59fa4007a"). InnerVolumeSpecName "kube-api-access-fjgbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:11:42.857809 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.857745    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a93444-064d-4e25-baae-06d59fa4007a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2a93444-064d-4e25-baae-06d59fa4007a" (UID: "f2a93444-064d-4e25-baae-06d59fa4007a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:11:42.917869 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.917837    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2a93444-064d-4e25-baae-06d59fa4007a" containerID="6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7" exitCode=0
Sep 17 19:11:42.918029 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.917886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" event={"ID":"f2a93444-064d-4e25-baae-06d59fa4007a","Type":"ContainerDied","Data":"6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7"}
Sep 17 19:11:42.918029 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.917894    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"
Sep 17 19:11:42.918029 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.917971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52" event={"ID":"f2a93444-064d-4e25-baae-06d59fa4007a","Type":"ContainerDied","Data":"36ce4ee7c094cb82f22a2fd30e94789bf95b9a4f3ef2fd38f256032bd9279b45"}
Sep 17 19:11:42.918029 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.918011    2577 scope.go:117] "RemoveContainer" containerID="bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41"
Sep 17 19:11:42.927358 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.927342    2577 scope.go:117] "RemoveContainer" containerID="6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7"
Sep 17 19:11:42.934699 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.934682    2577 scope.go:117] "RemoveContainer" containerID="cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad"
Sep 17 19:11:42.940383 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.940362    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"]
Sep 17 19:11:42.942234 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.942211    2577 scope.go:117] "RemoveContainer" containerID="bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41"
Sep 17 19:11:42.942597 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:11:42.942567    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41\": container with ID starting with bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41 not found: ID does not exist" containerID="bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41"
Sep 17 19:11:42.942677 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.942605    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41"} err="failed to get container status \"bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41\": rpc error: code = NotFound desc = could not find container \"bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41\": container with ID starting with bf39b836a17a2914202b4f8e87540d1f4c82ef740a9520e6c4c398a3d4ff9f41 not found: ID does not exist"
Sep 17 19:11:42.942677 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.942629    2577 scope.go:117] "RemoveContainer" containerID="6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7"
Sep 17 19:11:42.943025 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:11:42.942996    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7\": container with ID starting with 6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7 not found: ID does not exist" containerID="6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7"
Sep 17 19:11:42.943126 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.943032    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7"} err="failed to get container status \"6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7\": rpc error: code = NotFound desc = could not find container \"6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7\": container with ID starting with 6d0c4f24d5785a1dd94ab30806ee421554160c930d0a2b76f1721af8c86838a7 not found: ID does not exist"
Sep 17 19:11:42.943126 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.943051    2577 scope.go:117] "RemoveContainer" containerID="cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad"
Sep 17 19:11:42.943357 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:11:42.943310    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad\": container with ID starting with cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad not found: ID does not exist" containerID="cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad"
Sep 17 19:11:42.943357 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.943336    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad"} err="failed to get container status \"cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad\": rpc error: code = NotFound desc = could not find container \"cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad\": container with ID starting with cf6ffdbfe49d03b4f255e3e2eecb8c5259c3daf3f2938bd1a8336c55c79f37ad not found: ID does not exist"
Sep 17 19:11:42.945629 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.945607    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-xjp52"]
Sep 17 19:11:42.956573 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.956553    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjgbf\" (UniqueName: \"kubernetes.io/projected/f2a93444-064d-4e25-baae-06d59fa4007a-kube-api-access-fjgbf\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:11:42.956666 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.956576    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2a93444-064d-4e25-baae-06d59fa4007a-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:11:42.956666 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.956592    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:11:42.956666 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.956604    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2a93444-064d-4e25-baae-06d59fa4007a-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:11:42.956666 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:42.956616    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a93444-064d-4e25-baae-06d59fa4007a-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:11:43.238963 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.238938    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" path="/var/lib/kubelet/pods/f2a93444-064d-4e25-baae-06d59fa4007a/volumes"
Sep 17 19:11:43.834346 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.834310    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"]
Sep 17 19:11:43.834733 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.834672    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kube-rbac-proxy"
Sep 17 19:11:43.834733 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.834684    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kube-rbac-proxy"
Sep 17 19:11:43.834733 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.834695    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container"
Sep 17 19:11:43.834733 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.834700    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container"
Sep 17 19:11:43.834733 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.834710    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="storage-initializer"
Sep 17 19:11:43.834733 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.834716    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="storage-initializer"
Sep 17 19:11:43.834981 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.834779    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kube-rbac-proxy"
Sep 17 19:11:43.834981 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.834789    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a93444-064d-4e25-baae-06d59fa4007a" containerName="kserve-container"
Sep 17 19:11:43.839331 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.839314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:43.842171 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.842139    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 19:11:43.842306 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.842212    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 19:11:43.842306 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.842298    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:11:43.842425 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.842318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 19:11:43.842599 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.842582    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 19:11:43.842661 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.842624    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:11:43.851925 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.850988    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"]
Sep 17 19:11:43.964498 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.964457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:43.964687 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.964510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znht8\" (UniqueName: \"kubernetes.io/projected/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-api-access-znht8\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:43.964687 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.964590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e3855b-fd85-478a-9668-fd6990e7c151-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:43.964687 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:43.964635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7e3855b-fd85-478a-9668-fd6990e7c151-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:44.065295 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:44.065251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:44.065457 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:44.065312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znht8\" (UniqueName: \"kubernetes.io/projected/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-api-access-znht8\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:44.065457 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:44.065360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e3855b-fd85-478a-9668-fd6990e7c151-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:44.065457 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:44.065399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7e3855b-fd85-478a-9668-fd6990e7c151-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:44.065611 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:11:44.065528    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 19:11:44.065611 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:11:44.065595    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e3855b-fd85-478a-9668-fd6990e7c151-proxy-tls podName:a7e3855b-fd85-478a-9668-fd6990e7c151 nodeName:}" failed. No retries permitted until 2026-09-17 19:11:44.565579463 +0000 UTC m=+2539.611982343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a7e3855b-fd85-478a-9668-fd6990e7c151-proxy-tls") pod "isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" (UID: "a7e3855b-fd85-478a-9668-fd6990e7c151") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 19:11:44.065789 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:44.065768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7e3855b-fd85-478a-9668-fd6990e7c151-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:44.066046 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:44.066030    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:44.080052 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:44.080022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znht8\" (UniqueName: \"kubernetes.io/projected/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-api-access-znht8\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:44.570980 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:44.570946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e3855b-fd85-478a-9668-fd6990e7c151-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:44.573246 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:44.573212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e3855b-fd85-478a-9668-fd6990e7c151-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:44.750337 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:44.750286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:45.031313 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:45.031287    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"]
Sep 17 19:11:45.033674 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:11:45.033647    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e3855b_fd85_478a_9668_fd6990e7c151.slice/crio-f93d2c712bc66fa1c57dd6331ffad8b148597388864ad581c41285b1a29889da WatchSource:0}: Error finding container f93d2c712bc66fa1c57dd6331ffad8b148597388864ad581c41285b1a29889da: Status 404 returned error can't find the container with id f93d2c712bc66fa1c57dd6331ffad8b148597388864ad581c41285b1a29889da
Sep 17 19:11:45.934312 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:45.934274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" event={"ID":"a7e3855b-fd85-478a-9668-fd6990e7c151","Type":"ContainerStarted","Data":"4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f"}
Sep 17 19:11:45.934520 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:45.934337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" event={"ID":"a7e3855b-fd85-478a-9668-fd6990e7c151","Type":"ContainerStarted","Data":"f93d2c712bc66fa1c57dd6331ffad8b148597388864ad581c41285b1a29889da"}
Sep 17 19:11:46.941233 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:46.941199    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerID="4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f" exitCode=0
Sep 17 19:11:46.941626 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:46.941269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" event={"ID":"a7e3855b-fd85-478a-9668-fd6990e7c151","Type":"ContainerDied","Data":"4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f"}
Sep 17 19:11:47.948327 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:47.948293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" event={"ID":"a7e3855b-fd85-478a-9668-fd6990e7c151","Type":"ContainerStarted","Data":"9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3"}
Sep 17 19:11:47.948726 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:47.948350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" event={"ID":"a7e3855b-fd85-478a-9668-fd6990e7c151","Type":"ContainerStarted","Data":"c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9"}
Sep 17 19:11:47.948726 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:47.948428    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:47.988702 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:47.988646    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podStartSLOduration=3.896316638 podStartE2EDuration="4.988628091s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.092311453s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:47.985814173 +0000 UTC m=+2543.032217075" watchObservedRunningTime="2026-09-17 19:11:47.988628091 +0000 UTC m=+2543.035030993"
Sep 17 19:11:48.953897 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:48.953855    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:48.955053 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:48.955020    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:11:49.958678 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:49.958638    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:11:54.964950 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:54.964920    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:11:54.965418 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:11:54.965393    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:12:04.965764 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:12:04.965720    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:12:14.966167 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:12:14.966118    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:12:24.972136 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:12:24.972081    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:12:34.966374 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:12:34.966334    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:12:44.966227 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:12:44.966184    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:12:54.966117 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:12:54.966069    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:13:04.967118 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:04.967084    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:13:14.032448 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:14.032410    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"]
Sep 17 19:13:14.032950 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:14.032694    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kube-rbac-proxy" containerID="cri-o://9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3" gracePeriod=30
Sep 17 19:13:14.032950 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:14.032689    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" containerID="cri-o://c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9" gracePeriod=30
Sep 17 19:13:14.355159 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:14.355063    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerID="9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3" exitCode=2
Sep 17 19:13:14.355159 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:14.355136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" event={"ID":"a7e3855b-fd85-478a-9668-fd6990e7c151","Type":"ContainerDied","Data":"9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3"}
Sep 17 19:13:14.959843 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:14.959798    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 19:13:14.966253 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:14.966216    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:13:19.078894 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.078863    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:13:19.209411 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.209374    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znht8\" (UniqueName: \"kubernetes.io/projected/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-api-access-znht8\") pod \"a7e3855b-fd85-478a-9668-fd6990e7c151\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") "
Sep 17 19:13:19.209588 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.209449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e3855b-fd85-478a-9668-fd6990e7c151-proxy-tls\") pod \"a7e3855b-fd85-478a-9668-fd6990e7c151\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") "
Sep 17 19:13:19.209588 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.209470    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7e3855b-fd85-478a-9668-fd6990e7c151-kserve-provision-location\") pod \"a7e3855b-fd85-478a-9668-fd6990e7c151\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") "
Sep 17 19:13:19.209588 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.209537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-rbac-proxy-sar-config\") pod \"a7e3855b-fd85-478a-9668-fd6990e7c151\" (UID: \"a7e3855b-fd85-478a-9668-fd6990e7c151\") "
Sep 17 19:13:19.209879 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.209828    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e3855b-fd85-478a-9668-fd6990e7c151-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7e3855b-fd85-478a-9668-fd6990e7c151" (UID: "a7e3855b-fd85-478a-9668-fd6990e7c151"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:19.209879 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.209871    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7e3855b-fd85-478a-9668-fd6990e7c151" (UID: "a7e3855b-fd85-478a-9668-fd6990e7c151"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:19.211622 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.211601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e3855b-fd85-478a-9668-fd6990e7c151-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7e3855b-fd85-478a-9668-fd6990e7c151" (UID: "a7e3855b-fd85-478a-9668-fd6990e7c151"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:19.211727 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.211708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-api-access-znht8" (OuterVolumeSpecName: "kube-api-access-znht8") pod "a7e3855b-fd85-478a-9668-fd6990e7c151" (UID: "a7e3855b-fd85-478a-9668-fd6990e7c151"). InnerVolumeSpecName "kube-api-access-znht8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:19.310120 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.310085    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:13:19.310120 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.310117    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znht8\" (UniqueName: \"kubernetes.io/projected/a7e3855b-fd85-478a-9668-fd6990e7c151-kube-api-access-znht8\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:13:19.310120 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.310126    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e3855b-fd85-478a-9668-fd6990e7c151-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:13:19.310367 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.310134    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7e3855b-fd85-478a-9668-fd6990e7c151-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:13:19.381252 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.381220    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerID="c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9" exitCode=0
Sep 17 19:13:19.381422 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.381320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" event={"ID":"a7e3855b-fd85-478a-9668-fd6990e7c151","Type":"ContainerDied","Data":"c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9"}
Sep 17 19:13:19.381422 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.381364    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"
Sep 17 19:13:19.381422 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.381379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg" event={"ID":"a7e3855b-fd85-478a-9668-fd6990e7c151","Type":"ContainerDied","Data":"f93d2c712bc66fa1c57dd6331ffad8b148597388864ad581c41285b1a29889da"}
Sep 17 19:13:19.381422 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.381408    2577 scope.go:117] "RemoveContainer" containerID="9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3"
Sep 17 19:13:19.390266 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.390244    2577 scope.go:117] "RemoveContainer" containerID="c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9"
Sep 17 19:13:19.398165 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.398136    2577 scope.go:117] "RemoveContainer" containerID="4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f"
Sep 17 19:13:19.405855 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.405833    2577 scope.go:117] "RemoveContainer" containerID="9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3"
Sep 17 19:13:19.406229 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:13:19.406205    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3\": container with ID starting with 9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3 not found: ID does not exist" containerID="9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3"
Sep 17 19:13:19.406307 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.406242    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3"} err="failed to get container status \"9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3\": rpc error: code = NotFound desc = could not find container \"9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3\": container with ID starting with 9e914112f49bc7a570e50d035ee4d1dac5b5bf45e804cc35b5b837f9058a49c3 not found: ID does not exist"
Sep 17 19:13:19.406307 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.406267    2577 scope.go:117] "RemoveContainer" containerID="c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9"
Sep 17 19:13:19.406564 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:13:19.406538    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9\": container with ID starting with c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9 not found: ID does not exist" containerID="c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9"
Sep 17 19:13:19.406671 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.406570    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9"} err="failed to get container status \"c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9\": rpc error: code = NotFound desc = could not find container \"c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9\": container with ID starting with c003ef146896b139b336c60fda7ffdf9c8695bab5058b05c382202cbe5c7b2b9 not found: ID does not exist"
Sep 17 19:13:19.406671 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.406586    2577 scope.go:117] "RemoveContainer" containerID="4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f"
Sep 17 19:13:19.406884 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:13:19.406855    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f\": container with ID starting with 4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f not found: ID does not exist" containerID="4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f"
Sep 17 19:13:19.407007 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.406913    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f"} err="failed to get container status \"4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f\": rpc error: code = NotFound desc = could not find container \"4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f\": container with ID starting with 4eebc1088f007bf21b2716cdd6173980d4cae91ac70ff5a769c899111607408f not found: ID does not exist"
Sep 17 19:13:19.410839 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.410724    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"]
Sep 17 19:13:19.412894 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:19.412871    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fc9fb5bbd-j9fkg"]
Sep 17 19:13:21.239975 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:13:21.239942    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" path="/var/lib/kubelet/pods/a7e3855b-fd85-478a-9668-fd6990e7c151/volumes"
Sep 17 19:14:45.309130 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.309094    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"]
Sep 17 19:14:45.309552 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.309463    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kube-rbac-proxy"
Sep 17 19:14:45.309552 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.309475    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kube-rbac-proxy"
Sep 17 19:14:45.309552 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.309494    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container"
Sep 17 19:14:45.309552 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.309499    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container"
Sep 17 19:14:45.309552 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.309513    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="storage-initializer"
Sep 17 19:14:45.309552 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.309521    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="storage-initializer"
Sep 17 19:14:45.309745 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.309578    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kserve-container"
Sep 17 19:14:45.309745 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.309588    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e3855b-fd85-478a-9668-fd6990e7c151" containerName="kube-rbac-proxy"
Sep 17 19:14:45.312643 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.312626    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.315361 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.315337    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 19:14:45.315479 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.315338    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 19:14:45.315479 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.315371    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 19:14:45.315647 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.315632    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:14:45.316534 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.316512    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:14:45.316649 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.316527    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 19:14:45.323001 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.322979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"]
Sep 17 19:14:45.369213 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.369177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eb4370-81c0-4a31-ad84-0598f5fc7837-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.369390 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.369239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.369390 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.369291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.369390 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.369334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpq4v\" (UniqueName: \"kubernetes.io/projected/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-api-access-cpq4v\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.470416 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.470372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.470416 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.470425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.470679 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.470466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpq4v\" (UniqueName: \"kubernetes.io/projected/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-api-access-cpq4v\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.470679 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.470499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eb4370-81c0-4a31-ad84-0598f5fc7837-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.470894 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.470871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.471144 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.471122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.473107 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.473085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eb4370-81c0-4a31-ad84-0598f5fc7837-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.478831 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.478810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpq4v\" (UniqueName: \"kubernetes.io/projected/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-api-access-cpq4v\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.624765 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.624677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:45.898253 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:45.898224    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"]
Sep 17 19:14:45.900214 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:14:45.900186    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6eb4370_81c0_4a31_ad84_0598f5fc7837.slice/crio-fea40587e3c629b89e999a102a18757169a16f87792bb690f3cca2d4750f3acd WatchSource:0}: Error finding container fea40587e3c629b89e999a102a18757169a16f87792bb690f3cca2d4750f3acd: Status 404 returned error can't find the container with id fea40587e3c629b89e999a102a18757169a16f87792bb690f3cca2d4750f3acd
Sep 17 19:14:46.789174 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:46.789132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4" event={"ID":"b6eb4370-81c0-4a31-ad84-0598f5fc7837","Type":"ContainerStarted","Data":"174f7f4d3abeffe69e670dcf1409c8fcf09c87f1a90a2429951756e573a596c0"}
Sep 17 19:14:46.789678 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:46.789205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4" event={"ID":"b6eb4370-81c0-4a31-ad84-0598f5fc7837","Type":"ContainerStarted","Data":"fea40587e3c629b89e999a102a18757169a16f87792bb690f3cca2d4750f3acd"}
Sep 17 19:14:53.821849 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:53.821821    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4_b6eb4370-81c0-4a31-ad84-0598f5fc7837/storage-initializer/0.log"
Sep 17 19:14:53.822247 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:53.821862    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6eb4370-81c0-4a31-ad84-0598f5fc7837" containerID="174f7f4d3abeffe69e670dcf1409c8fcf09c87f1a90a2429951756e573a596c0" exitCode=1
Sep 17 19:14:53.822247 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:53.821935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4" event={"ID":"b6eb4370-81c0-4a31-ad84-0598f5fc7837","Type":"ContainerDied","Data":"174f7f4d3abeffe69e670dcf1409c8fcf09c87f1a90a2429951756e573a596c0"}
Sep 17 19:14:54.829510 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:54.829482    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4_b6eb4370-81c0-4a31-ad84-0598f5fc7837/storage-initializer/0.log"
Sep 17 19:14:54.829921 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:54.829543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4" event={"ID":"b6eb4370-81c0-4a31-ad84-0598f5fc7837","Type":"ContainerStarted","Data":"c8e73909a47ac4ca4fb2d5f21a4c242c53f6eba703c42b0f5a3b391c3ba94097"}
Sep 17 19:14:55.267351 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:55.267318    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"]
Sep 17 19:14:55.833997 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:55.833953    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4" podUID="b6eb4370-81c0-4a31-ad84-0598f5fc7837" containerName="storage-initializer" containerID="cri-o://c8e73909a47ac4ca4fb2d5f21a4c242c53f6eba703c42b0f5a3b391c3ba94097" gracePeriod=30
Sep 17 19:14:57.845300 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.845273    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4_b6eb4370-81c0-4a31-ad84-0598f5fc7837/storage-initializer/1.log"
Sep 17 19:14:57.845756 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.845741    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4_b6eb4370-81c0-4a31-ad84-0598f5fc7837/storage-initializer/0.log"
Sep 17 19:14:57.845814 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.845773    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6eb4370-81c0-4a31-ad84-0598f5fc7837" containerID="c8e73909a47ac4ca4fb2d5f21a4c242c53f6eba703c42b0f5a3b391c3ba94097" exitCode=1
Sep 17 19:14:57.845861 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.845810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4" event={"ID":"b6eb4370-81c0-4a31-ad84-0598f5fc7837","Type":"ContainerDied","Data":"c8e73909a47ac4ca4fb2d5f21a4c242c53f6eba703c42b0f5a3b391c3ba94097"}
Sep 17 19:14:57.845918 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.845864    2577 scope.go:117] "RemoveContainer" containerID="174f7f4d3abeffe69e670dcf1409c8fcf09c87f1a90a2429951756e573a596c0"
Sep 17 19:14:57.877665 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.877643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4_b6eb4370-81c0-4a31-ad84-0598f5fc7837/storage-initializer/1.log"
Sep 17 19:14:57.880763 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.880746    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:57.982678 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.982589    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpq4v\" (UniqueName: \"kubernetes.io/projected/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-api-access-cpq4v\") pod \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") "
Sep 17 19:14:57.982678 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.982637    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-rbac-proxy-sar-config\") pod \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") "
Sep 17 19:14:57.982889 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.982689    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eb4370-81c0-4a31-ad84-0598f5fc7837-proxy-tls\") pod \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") "
Sep 17 19:14:57.982889 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.982714    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kserve-provision-location\") pod \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\" (UID: \"b6eb4370-81c0-4a31-ad84-0598f5fc7837\") "
Sep 17 19:14:57.983055 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.983025    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6eb4370-81c0-4a31-ad84-0598f5fc7837" (UID: "b6eb4370-81c0-4a31-ad84-0598f5fc7837"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:57.983123 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.983076    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6eb4370-81c0-4a31-ad84-0598f5fc7837" (UID: "b6eb4370-81c0-4a31-ad84-0598f5fc7837"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:14:57.984862 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.984830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6eb4370-81c0-4a31-ad84-0598f5fc7837-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6eb4370-81c0-4a31-ad84-0598f5fc7837" (UID: "b6eb4370-81c0-4a31-ad84-0598f5fc7837"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:14:57.985014 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:57.984871    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-api-access-cpq4v" (OuterVolumeSpecName: "kube-api-access-cpq4v") pod "b6eb4370-81c0-4a31-ad84-0598f5fc7837" (UID: "b6eb4370-81c0-4a31-ad84-0598f5fc7837"). InnerVolumeSpecName "kube-api-access-cpq4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:58.083705 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:58.083653    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:14:58.083705 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:58.083698    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eb4370-81c0-4a31-ad84-0598f5fc7837-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:14:58.083705 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:58.083708    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:14:58.083705 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:58.083717    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpq4v\" (UniqueName: \"kubernetes.io/projected/b6eb4370-81c0-4a31-ad84-0598f5fc7837-kube-api-access-cpq4v\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:14:58.851898 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:58.851870    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4_b6eb4370-81c0-4a31-ad84-0598f5fc7837/storage-initializer/1.log"
Sep 17 19:14:58.852392 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:58.851987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4" event={"ID":"b6eb4370-81c0-4a31-ad84-0598f5fc7837","Type":"ContainerDied","Data":"fea40587e3c629b89e999a102a18757169a16f87792bb690f3cca2d4750f3acd"}
Sep 17 19:14:58.852392 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:58.852041    2577 scope.go:117] "RemoveContainer" containerID="c8e73909a47ac4ca4fb2d5f21a4c242c53f6eba703c42b0f5a3b391c3ba94097"
Sep 17 19:14:58.852392 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:58.852051    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"
Sep 17 19:14:58.891155 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:58.891125    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"]
Sep 17 19:14:58.895950 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:58.895918    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-56597d9f6d-797m4"]
Sep 17 19:14:59.239555 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:14:59.239517    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6eb4370-81c0-4a31-ad84-0598f5fc7837" path="/var/lib/kubelet/pods/b6eb4370-81c0-4a31-ad84-0598f5fc7837/volumes"
Sep 17 19:16:27.571892 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.571857    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"]
Sep 17 19:16:27.572406 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.572319    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6eb4370-81c0-4a31-ad84-0598f5fc7837" containerName="storage-initializer"
Sep 17 19:16:27.572406 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.572336    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6eb4370-81c0-4a31-ad84-0598f5fc7837" containerName="storage-initializer"
Sep 17 19:16:27.572406 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.572351    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6eb4370-81c0-4a31-ad84-0598f5fc7837" containerName="storage-initializer"
Sep 17 19:16:27.572406 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.572357    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6eb4370-81c0-4a31-ad84-0598f5fc7837" containerName="storage-initializer"
Sep 17 19:16:27.572607 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.572436    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6eb4370-81c0-4a31-ad84-0598f5fc7837" containerName="storage-initializer"
Sep 17 19:16:27.572607 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.572450    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6eb4370-81c0-4a31-ad84-0598f5fc7837" containerName="storage-initializer"
Sep 17 19:16:27.575499 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.575481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.578174 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.578148    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 19:16:27.578309 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.578224    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:16:27.578469 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.578452    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:16:27.579180 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.579165    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 19:16:27.579269 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.579179    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 19:16:27.579269 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.579169    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 19:16:27.586390 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.586365    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"]
Sep 17 19:16:27.648227 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.648192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598030ee-40a0-475c-aedf-a65c7299074c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.648227 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.648231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598030ee-40a0-475c-aedf-a65c7299074c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.648447 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.648251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4n57\" (UniqueName: \"kubernetes.io/projected/598030ee-40a0-475c-aedf-a65c7299074c-kube-api-access-k4n57\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.648447 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.648343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598030ee-40a0-475c-aedf-a65c7299074c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.749347 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.749311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598030ee-40a0-475c-aedf-a65c7299074c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.749518 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.749377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598030ee-40a0-475c-aedf-a65c7299074c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.749518 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.749394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598030ee-40a0-475c-aedf-a65c7299074c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.749518 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.749411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4n57\" (UniqueName: \"kubernetes.io/projected/598030ee-40a0-475c-aedf-a65c7299074c-kube-api-access-k4n57\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.749881 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.749860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598030ee-40a0-475c-aedf-a65c7299074c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.750044 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.750025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598030ee-40a0-475c-aedf-a65c7299074c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.751828 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.751808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598030ee-40a0-475c-aedf-a65c7299074c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.757825 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.757794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4n57\" (UniqueName: \"kubernetes.io/projected/598030ee-40a0-475c-aedf-a65c7299074c-kube-api-access-k4n57\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:27.886749 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:27.886657    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:28.165679 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:28.163884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"]
Sep 17 19:16:28.255238 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:28.255200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr" event={"ID":"598030ee-40a0-475c-aedf-a65c7299074c","Type":"ContainerStarted","Data":"38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689"}
Sep 17 19:16:28.255445 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:28.255269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr" event={"ID":"598030ee-40a0-475c-aedf-a65c7299074c","Type":"ContainerStarted","Data":"c51fbfdf4553ddff7b21ab87d98f4ba82a68d9733125d10be97096978d395b40"}
Sep 17 19:16:35.287358 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:35.287329    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr_598030ee-40a0-475c-aedf-a65c7299074c/storage-initializer/0.log"
Sep 17 19:16:35.287800 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:35.287369    2577 generic.go:356] "Generic (PLEG): container finished" podID="598030ee-40a0-475c-aedf-a65c7299074c" containerID="38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689" exitCode=1
Sep 17 19:16:35.287800 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:35.287402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr" event={"ID":"598030ee-40a0-475c-aedf-a65c7299074c","Type":"ContainerDied","Data":"38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689"}
Sep 17 19:16:36.293844 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:36.293812    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr_598030ee-40a0-475c-aedf-a65c7299074c/storage-initializer/0.log"
Sep 17 19:16:36.294260 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:36.293879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr" event={"ID":"598030ee-40a0-475c-aedf-a65c7299074c","Type":"ContainerStarted","Data":"cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494"}
Sep 17 19:16:37.543827 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:37.543792    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"]
Sep 17 19:16:37.544258 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:37.544034    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr" podUID="598030ee-40a0-475c-aedf-a65c7299074c" containerName="storage-initializer" containerID="cri-o://cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494" gracePeriod=30
Sep 17 19:16:38.691866 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.691846    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr_598030ee-40a0-475c-aedf-a65c7299074c/storage-initializer/1.log"
Sep 17 19:16:38.692288 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.692263    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr_598030ee-40a0-475c-aedf-a65c7299074c/storage-initializer/0.log"
Sep 17 19:16:38.694229 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.694214    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:38.700081 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.700030    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"]
Sep 17 19:16:38.700377 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.700365    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598030ee-40a0-475c-aedf-a65c7299074c" containerName="storage-initializer"
Sep 17 19:16:38.700425 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.700380    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="598030ee-40a0-475c-aedf-a65c7299074c" containerName="storage-initializer"
Sep 17 19:16:38.700425 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.700390    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598030ee-40a0-475c-aedf-a65c7299074c" containerName="storage-initializer"
Sep 17 19:16:38.700425 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.700396    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="598030ee-40a0-475c-aedf-a65c7299074c" containerName="storage-initializer"
Sep 17 19:16:38.700525 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.700465    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="598030ee-40a0-475c-aedf-a65c7299074c" containerName="storage-initializer"
Sep 17 19:16:38.700525 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.700473    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="598030ee-40a0-475c-aedf-a65c7299074c" containerName="storage-initializer"
Sep 17 19:16:38.703318 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.703303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.705837 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.705819    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 19:16:38.706009 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.705993    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:16:38.706477 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.706459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 19:16:38.717467 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.717443    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"]
Sep 17 19:16:38.748266 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.748237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4n57\" (UniqueName: \"kubernetes.io/projected/598030ee-40a0-475c-aedf-a65c7299074c-kube-api-access-k4n57\") pod \"598030ee-40a0-475c-aedf-a65c7299074c\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") "
Sep 17 19:16:38.748426 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.748310    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598030ee-40a0-475c-aedf-a65c7299074c-proxy-tls\") pod \"598030ee-40a0-475c-aedf-a65c7299074c\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") "
Sep 17 19:16:38.748426 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.748342    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598030ee-40a0-475c-aedf-a65c7299074c-kube-rbac-proxy-sar-config\") pod \"598030ee-40a0-475c-aedf-a65c7299074c\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") "
Sep 17 19:16:38.748426 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.748380    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598030ee-40a0-475c-aedf-a65c7299074c-kserve-provision-location\") pod \"598030ee-40a0-475c-aedf-a65c7299074c\" (UID: \"598030ee-40a0-475c-aedf-a65c7299074c\") "
Sep 17 19:16:38.748594 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.748502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdkpq\" (UniqueName: \"kubernetes.io/projected/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-api-access-pdkpq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.748594 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.748544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.748594 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.748570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.748767 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.748593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.748767 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.748698    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598030ee-40a0-475c-aedf-a65c7299074c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "598030ee-40a0-475c-aedf-a65c7299074c" (UID: "598030ee-40a0-475c-aedf-a65c7299074c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:38.749042 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.748646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.749144 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.749118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598030ee-40a0-475c-aedf-a65c7299074c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "598030ee-40a0-475c-aedf-a65c7299074c" (UID: "598030ee-40a0-475c-aedf-a65c7299074c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:16:38.756705 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.750701    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598030ee-40a0-475c-aedf-a65c7299074c-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:16:38.757036 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.756928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598030ee-40a0-475c-aedf-a65c7299074c-kube-api-access-k4n57" (OuterVolumeSpecName: "kube-api-access-k4n57") pod "598030ee-40a0-475c-aedf-a65c7299074c" (UID: "598030ee-40a0-475c-aedf-a65c7299074c"). InnerVolumeSpecName "kube-api-access-k4n57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:16:38.757201 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.757178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598030ee-40a0-475c-aedf-a65c7299074c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "598030ee-40a0-475c-aedf-a65c7299074c" (UID: "598030ee-40a0-475c-aedf-a65c7299074c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:16:38.851134 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.851134 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.851338 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.851338 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:16:38.851282    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 19:16:38.851338 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.851445 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:16:38.851352    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-proxy-tls podName:c6cec305-c98e-49d0-a2c3-c02fc6874c5b nodeName:}" failed. No retries permitted until 2026-09-17 19:16:39.351331306 +0000 UTC m=+2834.397734200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" (UID: "c6cec305-c98e-49d0-a2c3-c02fc6874c5b") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 19:16:38.851507 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdkpq\" (UniqueName: \"kubernetes.io/projected/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-api-access-pdkpq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.851557 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851540    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4n57\" (UniqueName: \"kubernetes.io/projected/598030ee-40a0-475c-aedf-a65c7299074c-kube-api-access-k4n57\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:16:38.851610 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851558    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598030ee-40a0-475c-aedf-a65c7299074c-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:16:38.851610 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851572    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598030ee-40a0-475c-aedf-a65c7299074c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:16:38.851707 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.851970 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.851970 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.851963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:38.860229 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:38.860209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdkpq\" (UniqueName: \"kubernetes.io/projected/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-api-access-pdkpq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:39.309382 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.309359    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr_598030ee-40a0-475c-aedf-a65c7299074c/storage-initializer/1.log"
Sep 17 19:16:39.309732 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.309717    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr_598030ee-40a0-475c-aedf-a65c7299074c/storage-initializer/0.log"
Sep 17 19:16:39.309797 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.309752    2577 generic.go:356] "Generic (PLEG): container finished" podID="598030ee-40a0-475c-aedf-a65c7299074c" containerID="cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494" exitCode=1
Sep 17 19:16:39.309797 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.309778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr" event={"ID":"598030ee-40a0-475c-aedf-a65c7299074c","Type":"ContainerDied","Data":"cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494"}
Sep 17 19:16:39.309919 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.309821    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr" event={"ID":"598030ee-40a0-475c-aedf-a65c7299074c","Type":"ContainerDied","Data":"c51fbfdf4553ddff7b21ab87d98f4ba82a68d9733125d10be97096978d395b40"}
Sep 17 19:16:39.309919 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.309829    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"
Sep 17 19:16:39.309919 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.309845    2577 scope.go:117] "RemoveContainer" containerID="cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494"
Sep 17 19:16:39.318402 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.318376    2577 scope.go:117] "RemoveContainer" containerID="38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689"
Sep 17 19:16:39.325832 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.325813    2577 scope.go:117] "RemoveContainer" containerID="cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494"
Sep 17 19:16:39.326120 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:16:39.326100    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494\": container with ID starting with cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494 not found: ID does not exist" containerID="cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494"
Sep 17 19:16:39.326173 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.326127    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494"} err="failed to get container status \"cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494\": rpc error: code = NotFound desc = could not find container \"cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494\": container with ID starting with cb33aaa94094be6fa11a01c430ee6ca58d0577769fbfe9f9679fb0f9a6119494 not found: ID does not exist"
Sep 17 19:16:39.326173 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.326144    2577 scope.go:117] "RemoveContainer" containerID="38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689"
Sep 17 19:16:39.326431 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:16:39.326414    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689\": container with ID starting with 38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689 not found: ID does not exist" containerID="38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689"
Sep 17 19:16:39.326494 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.326436    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689"} err="failed to get container status \"38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689\": rpc error: code = NotFound desc = could not find container \"38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689\": container with ID starting with 38fb09c742f64ad85237c04bc6ddf018cf1f1ee645b687ffe03368d824185689 not found: ID does not exist"
Sep 17 19:16:39.343965 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.343934    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"]
Sep 17 19:16:39.348975 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.348945    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cfcbffbcb-hq9cr"]
Sep 17 19:16:39.356622 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.356602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:39.358978 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.358957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:39.613059 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.613027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:39.890843 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:39.890817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"]
Sep 17 19:16:39.892788 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:16:39.892759    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6cec305_c98e_49d0_a2c3_c02fc6874c5b.slice/crio-a2e0df87e9d9e2d19c6bafca7ea234ae5cc5fc0db152953c04cf5a818141c4a5 WatchSource:0}: Error finding container a2e0df87e9d9e2d19c6bafca7ea234ae5cc5fc0db152953c04cf5a818141c4a5: Status 404 returned error can't find the container with id a2e0df87e9d9e2d19c6bafca7ea234ae5cc5fc0db152953c04cf5a818141c4a5
Sep 17 19:16:40.319199 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:40.319150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" event={"ID":"c6cec305-c98e-49d0-a2c3-c02fc6874c5b","Type":"ContainerStarted","Data":"2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad"}
Sep 17 19:16:40.319363 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:40.319218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" event={"ID":"c6cec305-c98e-49d0-a2c3-c02fc6874c5b","Type":"ContainerStarted","Data":"a2e0df87e9d9e2d19c6bafca7ea234ae5cc5fc0db152953c04cf5a818141c4a5"}
Sep 17 19:16:41.239243 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:41.239160    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="598030ee-40a0-475c-aedf-a65c7299074c" path="/var/lib/kubelet/pods/598030ee-40a0-475c-aedf-a65c7299074c/volumes"
Sep 17 19:16:41.324607 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:41.324575    2577 generic.go:356] "Generic (PLEG): container finished" podID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerID="2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad" exitCode=0
Sep 17 19:16:41.324780 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:41.324617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" event={"ID":"c6cec305-c98e-49d0-a2c3-c02fc6874c5b","Type":"ContainerDied","Data":"2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad"}
Sep 17 19:16:42.331878 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:42.331843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" event={"ID":"c6cec305-c98e-49d0-a2c3-c02fc6874c5b","Type":"ContainerStarted","Data":"a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8"}
Sep 17 19:16:42.332362 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:42.331931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" event={"ID":"c6cec305-c98e-49d0-a2c3-c02fc6874c5b","Type":"ContainerStarted","Data":"2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782"}
Sep 17 19:16:42.332362 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:42.331966    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:42.332362 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:42.332078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:42.333597 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:42.333573    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 19:16:42.357536 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:42.357489    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podStartSLOduration=3.330735804 podStartE2EDuration="4.357473079s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.026737275s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:42.356079263 +0000 UTC m=+2837.402482183" watchObservedRunningTime="2026-09-17 19:16:42.357473079 +0000 UTC m=+2837.403875980"
Sep 17 19:16:43.337366 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:43.337325    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 19:16:48.342054 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:48.342023    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:16:48.342570 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:48.342537    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 19:16:58.342757 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:16:58.342715    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 19:17:08.342637 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:08.342544    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 19:17:18.343391 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:18.343347    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 19:17:28.342856 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:28.342813    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 19:17:38.342612 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:38.342560    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 19:17:48.343127 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:48.343084    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 19:17:56.234152 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:56.234114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:17:58.695848 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:58.695815    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"]
Sep 17 19:17:58.696309 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:58.696066    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container" containerID="cri-o://2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782" gracePeriod=30
Sep 17 19:17:58.696309 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:58.696109    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kube-rbac-proxy" containerID="cri-o://a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8" gracePeriod=30
Sep 17 19:17:59.711659 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.711621    2577 generic.go:356] "Generic (PLEG): container finished" podID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerID="a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8" exitCode=2
Sep 17 19:17:59.712051 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.711695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" event={"ID":"c6cec305-c98e-49d0-a2c3-c02fc6874c5b","Type":"ContainerDied","Data":"a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8"}
Sep 17 19:17:59.825687 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.825654    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"]
Sep 17 19:17:59.829405 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.829387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:17:59.832588 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.832562    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 19:17:59.832712 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.832587    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 19:17:59.842630 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.842602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"]
Sep 17 19:17:59.963397 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.963304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f4fbefb-54a8-45e3-855a-248c85ba0a74-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:17:59.963397 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.963370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:17:59.963635 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.963434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwjt\" (UniqueName: \"kubernetes.io/projected/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-api-access-jgwjt\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:17:59.963635 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:17:59.963521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:00.064846 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.064809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:00.065051 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.064853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f4fbefb-54a8-45e3-855a-248c85ba0a74-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:00.065051 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.064888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:00.065051 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.064952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwjt\" (UniqueName: \"kubernetes.io/projected/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-api-access-jgwjt\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:00.065352 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.065329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:00.065523 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.065503    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:00.067378 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.067354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f4fbefb-54a8-45e3-855a-248c85ba0a74-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:00.073866 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.073842    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwjt\" (UniqueName: \"kubernetes.io/projected/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-api-access-jgwjt\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:00.139020 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.138982    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:00.416688 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.416661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"]
Sep 17 19:18:00.418392 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:18:00.418368    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4fbefb_54a8_45e3_855a_248c85ba0a74.slice/crio-4cd8989c6fd9c48ef52a4616df1ce479c51aad7d5473252801cd55aa132d8fd3 WatchSource:0}: Error finding container 4cd8989c6fd9c48ef52a4616df1ce479c51aad7d5473252801cd55aa132d8fd3: Status 404 returned error can't find the container with id 4cd8989c6fd9c48ef52a4616df1ce479c51aad7d5473252801cd55aa132d8fd3
Sep 17 19:18:00.718961 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.718923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt" event={"ID":"4f4fbefb-54a8-45e3-855a-248c85ba0a74","Type":"ContainerStarted","Data":"17cc698bb51d4fb1f4acba386f76774e1584f99895dd05a7f762072ed884ef95"}
Sep 17 19:18:00.719425 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:00.718994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt" event={"ID":"4f4fbefb-54a8-45e3-855a-248c85ba0a74","Type":"ContainerStarted","Data":"4cd8989c6fd9c48ef52a4616df1ce479c51aad7d5473252801cd55aa132d8fd3"}
Sep 17 19:18:03.338033 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.337988    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Sep 17 19:18:03.539373 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.539348    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:18:03.698837 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.698805    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-cabundle-cert\") pod \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") "
Sep 17 19:18:03.699048 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.698873    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-rbac-proxy-sar-config\") pod \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") "
Sep 17 19:18:03.699048 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.698990    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdkpq\" (UniqueName: \"kubernetes.io/projected/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-api-access-pdkpq\") pod \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") "
Sep 17 19:18:03.699048 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.699020    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kserve-provision-location\") pod \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") "
Sep 17 19:18:03.699166 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.699057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-proxy-tls\") pod \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\" (UID: \"c6cec305-c98e-49d0-a2c3-c02fc6874c5b\") "
Sep 17 19:18:03.699356 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.699330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6cec305-c98e-49d0-a2c3-c02fc6874c5b" (UID: "c6cec305-c98e-49d0-a2c3-c02fc6874c5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:03.699399 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.699330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6cec305-c98e-49d0-a2c3-c02fc6874c5b" (UID: "c6cec305-c98e-49d0-a2c3-c02fc6874c5b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:18:03.699399 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.699276    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6cec305-c98e-49d0-a2c3-c02fc6874c5b" (UID: "c6cec305-c98e-49d0-a2c3-c02fc6874c5b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:18:03.701131 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.701108    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-api-access-pdkpq" (OuterVolumeSpecName: "kube-api-access-pdkpq") pod "c6cec305-c98e-49d0-a2c3-c02fc6874c5b" (UID: "c6cec305-c98e-49d0-a2c3-c02fc6874c5b"). InnerVolumeSpecName "kube-api-access-pdkpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:18:03.701235 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.701221    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6cec305-c98e-49d0-a2c3-c02fc6874c5b" (UID: "c6cec305-c98e-49d0-a2c3-c02fc6874c5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:18:03.734463 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.734433    2577 generic.go:356] "Generic (PLEG): container finished" podID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerID="2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782" exitCode=0
Sep 17 19:18:03.734579 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.734485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" event={"ID":"c6cec305-c98e-49d0-a2c3-c02fc6874c5b","Type":"ContainerDied","Data":"2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782"}
Sep 17 19:18:03.734579 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.734494    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"
Sep 17 19:18:03.734579 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.734540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j" event={"ID":"c6cec305-c98e-49d0-a2c3-c02fc6874c5b","Type":"ContainerDied","Data":"a2e0df87e9d9e2d19c6bafca7ea234ae5cc5fc0db152953c04cf5a818141c4a5"}
Sep 17 19:18:03.734579 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.734570    2577 scope.go:117] "RemoveContainer" containerID="a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8"
Sep 17 19:18:03.743201 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.743182    2577 scope.go:117] "RemoveContainer" containerID="2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782"
Sep 17 19:18:03.750593 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.750578    2577 scope.go:117] "RemoveContainer" containerID="2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad"
Sep 17 19:18:03.758017 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.758000    2577 scope.go:117] "RemoveContainer" containerID="a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8"
Sep 17 19:18:03.758265 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:18:03.758248    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8\": container with ID starting with a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8 not found: ID does not exist" containerID="a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8"
Sep 17 19:18:03.758342 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.758272    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8"} err="failed to get container status \"a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8\": rpc error: code = NotFound desc = could not find container \"a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8\": container with ID starting with a7b329b9e426a20a9e5e629d0d1009c8c77831562ae47fc8a66457662c340bc8 not found: ID does not exist"
Sep 17 19:18:03.758342 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.758287    2577 scope.go:117] "RemoveContainer" containerID="2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782"
Sep 17 19:18:03.758535 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:18:03.758517    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782\": container with ID starting with 2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782 not found: ID does not exist" containerID="2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782"
Sep 17 19:18:03.758577 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.758545    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782"} err="failed to get container status \"2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782\": rpc error: code = NotFound desc = could not find container \"2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782\": container with ID starting with 2ab506549f26dc55314de6b2a087af8d60b4ae21f61f182a8cd6f4312e531782 not found: ID does not exist"
Sep 17 19:18:03.758577 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.758562    2577 scope.go:117] "RemoveContainer" containerID="2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad"
Sep 17 19:18:03.758801 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:18:03.758785    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad\": container with ID starting with 2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad not found: ID does not exist" containerID="2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad"
Sep 17 19:18:03.758857 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.758804    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad"} err="failed to get container status \"2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad\": rpc error: code = NotFound desc = could not find container \"2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad\": container with ID starting with 2f1230a19fc461a355b7042cc4fabb3ffbcd29ccc903a399c4b7677cb9f636ad not found: ID does not exist"
Sep 17 19:18:03.770232 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.770209    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"]
Sep 17 19:18:03.776766 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.776745    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-869cbf675f-6695j"]
Sep 17 19:18:03.799935 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.799893    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.799935 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.799926    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdkpq\" (UniqueName: \"kubernetes.io/projected/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kube-api-access-pdkpq\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.799935 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.799936    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.800092 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.799945    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.800092 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:03.799954    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6cec305-c98e-49d0-a2c3-c02fc6874c5b-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:18:04.741501 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:04.741473    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt_4f4fbefb-54a8-45e3-855a-248c85ba0a74/storage-initializer/0.log"
Sep 17 19:18:04.741980 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:04.741509    2577 generic.go:356] "Generic (PLEG): container finished" podID="4f4fbefb-54a8-45e3-855a-248c85ba0a74" containerID="17cc698bb51d4fb1f4acba386f76774e1584f99895dd05a7f762072ed884ef95" exitCode=1
Sep 17 19:18:04.741980 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:04.741537    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt" event={"ID":"4f4fbefb-54a8-45e3-855a-248c85ba0a74","Type":"ContainerDied","Data":"17cc698bb51d4fb1f4acba386f76774e1584f99895dd05a7f762072ed884ef95"}
Sep 17 19:18:05.238725 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:05.238685    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" path="/var/lib/kubelet/pods/c6cec305-c98e-49d0-a2c3-c02fc6874c5b/volumes"
Sep 17 19:18:05.748480 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:05.748457    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt_4f4fbefb-54a8-45e3-855a-248c85ba0a74/storage-initializer/0.log"
Sep 17 19:18:05.748865 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:05.748533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt" event={"ID":"4f4fbefb-54a8-45e3-855a-248c85ba0a74","Type":"ContainerStarted","Data":"430fa6205485736ab31e5d2502bd7a66fe5e804ec08b9ce829c3b2667a94429d"}
Sep 17 19:18:08.764362 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:08.764335    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt_4f4fbefb-54a8-45e3-855a-248c85ba0a74/storage-initializer/1.log"
Sep 17 19:18:08.764747 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:08.764724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt_4f4fbefb-54a8-45e3-855a-248c85ba0a74/storage-initializer/0.log"
Sep 17 19:18:08.764812 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:08.764758    2577 generic.go:356] "Generic (PLEG): container finished" podID="4f4fbefb-54a8-45e3-855a-248c85ba0a74" containerID="430fa6205485736ab31e5d2502bd7a66fe5e804ec08b9ce829c3b2667a94429d" exitCode=1
Sep 17 19:18:08.764856 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:08.764841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt" event={"ID":"4f4fbefb-54a8-45e3-855a-248c85ba0a74","Type":"ContainerDied","Data":"430fa6205485736ab31e5d2502bd7a66fe5e804ec08b9ce829c3b2667a94429d"}
Sep 17 19:18:08.764933 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:08.764919    2577 scope.go:117] "RemoveContainer" containerID="17cc698bb51d4fb1f4acba386f76774e1584f99895dd05a7f762072ed884ef95"
Sep 17 19:18:08.765300 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:08.765278    2577 scope.go:117] "RemoveContainer" containerID="17cc698bb51d4fb1f4acba386f76774e1584f99895dd05a7f762072ed884ef95"
Sep 17 19:18:08.776568 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:18:08.776542    2577 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt_kserve-ci-e2e-test_4f4fbefb-54a8-45e3-855a-248c85ba0a74_0 in pod sandbox 4cd8989c6fd9c48ef52a4616df1ce479c51aad7d5473252801cd55aa132d8fd3 from index: no such id: '17cc698bb51d4fb1f4acba386f76774e1584f99895dd05a7f762072ed884ef95'" containerID="17cc698bb51d4fb1f4acba386f76774e1584f99895dd05a7f762072ed884ef95"
Sep 17 19:18:08.776727 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:08.776626    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cc698bb51d4fb1f4acba386f76774e1584f99895dd05a7f762072ed884ef95"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt_kserve-ci-e2e-test_4f4fbefb-54a8-45e3-855a-248c85ba0a74_0 in pod sandbox 4cd8989c6fd9c48ef52a4616df1ce479c51aad7d5473252801cd55aa132d8fd3 from index: no such id: '17cc698bb51d4fb1f4acba386f76774e1584f99895dd05a7f762072ed884ef95'"
Sep 17 19:18:08.776727 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:18:08.776547    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt_kserve-ci-e2e-test(4f4fbefb-54a8-45e3-855a-248c85ba0a74)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt" podUID="4f4fbefb-54a8-45e3-855a-248c85ba0a74"
Sep 17 19:18:09.771567 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.771542    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt_4f4fbefb-54a8-45e3-855a-248c85ba0a74/storage-initializer/1.log"
Sep 17 19:18:09.775120 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.775088    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"]
Sep 17 19:18:09.906583 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.906558    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt_4f4fbefb-54a8-45e3-855a-248c85ba0a74/storage-initializer/1.log"
Sep 17 19:18:09.910176 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.910159    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:09.947504 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.947476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kserve-provision-location\") pod \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") "
Sep 17 19:18:09.947698 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.947531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgwjt\" (UniqueName: \"kubernetes.io/projected/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-api-access-jgwjt\") pod \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") "
Sep 17 19:18:09.947698 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.947564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f4fbefb-54a8-45e3-855a-248c85ba0a74-proxy-tls\") pod \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") "
Sep 17 19:18:09.947698 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.947586    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-rbac-proxy-sar-config\") pod \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\" (UID: \"4f4fbefb-54a8-45e3-855a-248c85ba0a74\") "
Sep 17 19:18:09.947868 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.947779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f4fbefb-54a8-45e3-855a-248c85ba0a74" (UID: "4f4fbefb-54a8-45e3-855a-248c85ba0a74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:09.947949 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.947888    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:18:09.948039 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.948019    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f4fbefb-54a8-45e3-855a-248c85ba0a74" (UID: "4f4fbefb-54a8-45e3-855a-248c85ba0a74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:18:09.949660 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.949643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4fbefb-54a8-45e3-855a-248c85ba0a74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f4fbefb-54a8-45e3-855a-248c85ba0a74" (UID: "4f4fbefb-54a8-45e3-855a-248c85ba0a74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:18:09.949713 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:09.949689    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-api-access-jgwjt" (OuterVolumeSpecName: "kube-api-access-jgwjt") pod "4f4fbefb-54a8-45e3-855a-248c85ba0a74" (UID: "4f4fbefb-54a8-45e3-855a-248c85ba0a74"). InnerVolumeSpecName "kube-api-access-jgwjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:18:10.049014 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.048934    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgwjt\" (UniqueName: \"kubernetes.io/projected/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-api-access-jgwjt\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:18:10.049014 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.048961    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f4fbefb-54a8-45e3-855a-248c85ba0a74-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:18:10.049014 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.048970    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f4fbefb-54a8-45e3-855a-248c85ba0a74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:18:10.778197 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.778163    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt_4f4fbefb-54a8-45e3-855a-248c85ba0a74/storage-initializer/1.log"
Sep 17 19:18:10.778646 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.778267    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"
Sep 17 19:18:10.778646 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.778306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt" event={"ID":"4f4fbefb-54a8-45e3-855a-248c85ba0a74","Type":"ContainerDied","Data":"4cd8989c6fd9c48ef52a4616df1ce479c51aad7d5473252801cd55aa132d8fd3"}
Sep 17 19:18:10.778646 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.778392    2577 scope.go:117] "RemoveContainer" containerID="430fa6205485736ab31e5d2502bd7a66fe5e804ec08b9ce829c3b2667a94429d"
Sep 17 19:18:10.822890 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.822860    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"]
Sep 17 19:18:10.826475 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.826443    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-549f55cfb9-mlgmt"]
Sep 17 19:18:10.933209 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933176    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"]
Sep 17 19:18:10.933540 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933527    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f4fbefb-54a8-45e3-855a-248c85ba0a74" containerName="storage-initializer"
Sep 17 19:18:10.933584 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933541    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f4fbefb-54a8-45e3-855a-248c85ba0a74" containerName="storage-initializer"
Sep 17 19:18:10.933584 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933552    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="storage-initializer"
Sep 17 19:18:10.933584 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933557    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="storage-initializer"
Sep 17 19:18:10.933584 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933565    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f4fbefb-54a8-45e3-855a-248c85ba0a74" containerName="storage-initializer"
Sep 17 19:18:10.933584 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933571    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f4fbefb-54a8-45e3-855a-248c85ba0a74" containerName="storage-initializer"
Sep 17 19:18:10.933736 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933589    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kube-rbac-proxy"
Sep 17 19:18:10.933736 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933595    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kube-rbac-proxy"
Sep 17 19:18:10.933736 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933601    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container"
Sep 17 19:18:10.933736 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933606    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container"
Sep 17 19:18:10.933736 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933656    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f4fbefb-54a8-45e3-855a-248c85ba0a74" containerName="storage-initializer"
Sep 17 19:18:10.933736 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933665    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kube-rbac-proxy"
Sep 17 19:18:10.933736 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933671    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f4fbefb-54a8-45e3-855a-248c85ba0a74" containerName="storage-initializer"
Sep 17 19:18:10.933736 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.933681    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6cec305-c98e-49d0-a2c3-c02fc6874c5b" containerName="kserve-container"
Sep 17 19:18:10.938148 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.938124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:10.943070 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.943045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-7e891e-predictor-serving-cert\""
Sep 17 19:18:10.943207 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.943149    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:18:10.943278 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.943223    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:18:10.943532 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.943513    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-7e891e-kube-rbac-proxy-sar-config\""
Sep 17 19:18:10.943632 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.943589    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:18:10.944034 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.944010    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:18:10.945501 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.945482    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\""
Sep 17 19:18:10.955467 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.955436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a2009ae-228b-4928-ba95-860cc502c6ff-proxy-tls\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:10.955580 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.955485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-cabundle-cert\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:10.955580 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.955513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnvl\" (UniqueName: \"kubernetes.io/projected/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-api-access-2vnvl\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:10.955580 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.955565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a2009ae-228b-4928-ba95-860cc502c6ff-kserve-provision-location\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:10.955699 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.955615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-rbac-proxy-sar-config\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:10.961392 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:10.961370    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"]
Sep 17 19:18:11.056142 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.056060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a2009ae-228b-4928-ba95-860cc502c6ff-proxy-tls\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.056142 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.056123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-cabundle-cert\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.056372 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.056152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnvl\" (UniqueName: \"kubernetes.io/projected/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-api-access-2vnvl\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.056372 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.056169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a2009ae-228b-4928-ba95-860cc502c6ff-kserve-provision-location\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.056372 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.056188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-rbac-proxy-sar-config\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.056680 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.056656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a2009ae-228b-4928-ba95-860cc502c6ff-kserve-provision-location\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.056828 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.056807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-rbac-proxy-sar-config\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.056867 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.056852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-cabundle-cert\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.058521 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.058504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a2009ae-228b-4928-ba95-860cc502c6ff-proxy-tls\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.065550 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.065527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnvl\" (UniqueName: \"kubernetes.io/projected/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-api-access-2vnvl\") pod \"isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") " pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.239023 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.238978    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f4fbefb-54a8-45e3-855a-248c85ba0a74" path="/var/lib/kubelet/pods/4f4fbefb-54a8-45e3-855a-248c85ba0a74/volumes"
Sep 17 19:18:11.248132 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.248107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:11.524061 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.523999    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"]
Sep 17 19:18:11.526425 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:18:11.526399    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2009ae_228b_4928_ba95_860cc502c6ff.slice/crio-35229e0a2e1dadc6670c32ea9d1ba9d13417aba40427e0ef9100f6cf79198635 WatchSource:0}: Error finding container 35229e0a2e1dadc6670c32ea9d1ba9d13417aba40427e0ef9100f6cf79198635: Status 404 returned error can't find the container with id 35229e0a2e1dadc6670c32ea9d1ba9d13417aba40427e0ef9100f6cf79198635
Sep 17 19:18:11.786020 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.785896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" event={"ID":"1a2009ae-228b-4928-ba95-860cc502c6ff","Type":"ContainerStarted","Data":"5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93"}
Sep 17 19:18:11.786020 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:11.785997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" event={"ID":"1a2009ae-228b-4928-ba95-860cc502c6ff","Type":"ContainerStarted","Data":"35229e0a2e1dadc6670c32ea9d1ba9d13417aba40427e0ef9100f6cf79198635"}
Sep 17 19:18:15.806219 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:15.806184    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerID="5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93" exitCode=0
Sep 17 19:18:15.806582 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:15.806252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" event={"ID":"1a2009ae-228b-4928-ba95-860cc502c6ff","Type":"ContainerDied","Data":"5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93"}
Sep 17 19:18:16.813340 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:16.813299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" event={"ID":"1a2009ae-228b-4928-ba95-860cc502c6ff","Type":"ContainerStarted","Data":"0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647"}
Sep 17 19:18:16.813798 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:16.813409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" event={"ID":"1a2009ae-228b-4928-ba95-860cc502c6ff","Type":"ContainerStarted","Data":"9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee"}
Sep 17 19:18:16.813798 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:16.813639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:17.818045 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:17.818014    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:17.819336 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:17.819309    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:18:18.823088 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:18.823046    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:18:23.827225 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:23.827198    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:18:23.827680 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:23.827652    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:18:23.858546 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:23.858499    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podStartSLOduration=10.232830961 podStartE2EDuration="13.858485616s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.625654655s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:18:16.864339427 +0000 UTC m=+2931.910742360" watchObservedRunningTime="2026-09-17 19:18:23.858485616 +0000 UTC m=+2938.904888518"
Sep 17 19:18:33.828425 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:33.828386    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:18:43.827925 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:43.827864    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:18:53.828046 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:18:53.828003    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:19:03.827826 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:19:03.827778    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:19:13.828661 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:19:13.828617    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:19:23.828679 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:19:23.828633    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:19:28.233952 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:19:28.233884    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:19:38.235077 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:19:38.235045    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:20:12.178068 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:12.177989    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"]
Sep 17 19:20:12.178633 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:12.178252    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container" containerID="cri-o://9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee" gracePeriod=30
Sep 17 19:20:12.178633 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:12.178300    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kube-rbac-proxy" containerID="cri-o://0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647" gracePeriod=30
Sep 17 19:20:12.344136 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:12.344104    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerID="0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647" exitCode=2
Sep 17 19:20:12.344287 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:12.344176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" event={"ID":"1a2009ae-228b-4928-ba95-860cc502c6ff","Type":"ContainerDied","Data":"0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647"}
Sep 17 19:20:13.824028 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:13.823982    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 17 19:20:17.130991 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.130963    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:20:17.285026 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.284947    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a2009ae-228b-4928-ba95-860cc502c6ff-proxy-tls\") pod \"1a2009ae-228b-4928-ba95-860cc502c6ff\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") "
Sep 17 19:20:17.285026 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.284982    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a2009ae-228b-4928-ba95-860cc502c6ff-kserve-provision-location\") pod \"1a2009ae-228b-4928-ba95-860cc502c6ff\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") "
Sep 17 19:20:17.285026 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.285020    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-cabundle-cert\") pod \"1a2009ae-228b-4928-ba95-860cc502c6ff\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") "
Sep 17 19:20:17.285300 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.285148    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-rbac-proxy-sar-config\") pod \"1a2009ae-228b-4928-ba95-860cc502c6ff\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") "
Sep 17 19:20:17.285300 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.285210    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vnvl\" (UniqueName: \"kubernetes.io/projected/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-api-access-2vnvl\") pod \"1a2009ae-228b-4928-ba95-860cc502c6ff\" (UID: \"1a2009ae-228b-4928-ba95-860cc502c6ff\") "
Sep 17 19:20:17.285428 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.285402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2009ae-228b-4928-ba95-860cc502c6ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a2009ae-228b-4928-ba95-860cc502c6ff" (UID: "1a2009ae-228b-4928-ba95-860cc502c6ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:20:17.285478 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.285432    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a2009ae-228b-4928-ba95-860cc502c6ff" (UID: "1a2009ae-228b-4928-ba95-860cc502c6ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:17.285527 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.285473    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a2009ae-228b-4928-ba95-860cc502c6ff" (UID: "1a2009ae-228b-4928-ba95-860cc502c6ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:17.285579 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.285562    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a2009ae-228b-4928-ba95-860cc502c6ff-kserve-provision-location\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:20:17.285619 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.285581    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-cabundle-cert\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:20:17.285619 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.285591    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:20:17.287171 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.287149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-api-access-2vnvl" (OuterVolumeSpecName: "kube-api-access-2vnvl") pod "1a2009ae-228b-4928-ba95-860cc502c6ff" (UID: "1a2009ae-228b-4928-ba95-860cc502c6ff"). InnerVolumeSpecName "kube-api-access-2vnvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:20:17.287267 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.287177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2009ae-228b-4928-ba95-860cc502c6ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a2009ae-228b-4928-ba95-860cc502c6ff" (UID: "1a2009ae-228b-4928-ba95-860cc502c6ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:20:17.370489 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.370460    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerID="9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee" exitCode=0
Sep 17 19:20:17.370676 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.370519    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"
Sep 17 19:20:17.370676 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.370551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" event={"ID":"1a2009ae-228b-4928-ba95-860cc502c6ff","Type":"ContainerDied","Data":"9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee"}
Sep 17 19:20:17.370676 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.370624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr" event={"ID":"1a2009ae-228b-4928-ba95-860cc502c6ff","Type":"ContainerDied","Data":"35229e0a2e1dadc6670c32ea9d1ba9d13417aba40427e0ef9100f6cf79198635"}
Sep 17 19:20:17.370676 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.370662    2577 scope.go:117] "RemoveContainer" containerID="0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647"
Sep 17 19:20:17.379720 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.379704    2577 scope.go:117] "RemoveContainer" containerID="9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee"
Sep 17 19:20:17.387618 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.387564    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vnvl\" (UniqueName: \"kubernetes.io/projected/1a2009ae-228b-4928-ba95-860cc502c6ff-kube-api-access-2vnvl\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:20:17.387618 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.387595    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a2009ae-228b-4928-ba95-860cc502c6ff-proxy-tls\") on node \"ip-10-0-131-128.ec2.internal\" DevicePath \"\""
Sep 17 19:20:17.389671 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.389652    2577 scope.go:117] "RemoveContainer" containerID="5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93"
Sep 17 19:20:17.397220 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.397195    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"]
Sep 17 19:20:17.397443 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.397431    2577 scope.go:117] "RemoveContainer" containerID="0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647"
Sep 17 19:20:17.397697 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:20:17.397681    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647\": container with ID starting with 0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647 not found: ID does not exist" containerID="0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647"
Sep 17 19:20:17.397741 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.397704    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647"} err="failed to get container status \"0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647\": rpc error: code = NotFound desc = could not find container \"0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647\": container with ID starting with 0aca017ed0550279c5ec4f0ba6dfab3a9e19f10bcde1dd6e604f183a2b879647 not found: ID does not exist"
Sep 17 19:20:17.397741 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.397720    2577 scope.go:117] "RemoveContainer" containerID="9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee"
Sep 17 19:20:17.398015 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:20:17.397986    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee\": container with ID starting with 9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee not found: ID does not exist" containerID="9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee"
Sep 17 19:20:17.398062 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.398013    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee"} err="failed to get container status \"9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee\": rpc error: code = NotFound desc = could not find container \"9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee\": container with ID starting with 9bdcf11e41bab055d7bc4c3b3d30305dd36ff22c642edaca3334c6a861861dee not found: ID does not exist"
Sep 17 19:20:17.398062 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.398029    2577 scope.go:117] "RemoveContainer" containerID="5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93"
Sep 17 19:20:17.398270 ip-10-0-131-128 kubenswrapper[2577]: E0917 19:20:17.398254    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93\": container with ID starting with 5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93 not found: ID does not exist" containerID="5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93"
Sep 17 19:20:17.398317 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.398274    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93"} err="failed to get container status \"5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93\": rpc error: code = NotFound desc = could not find container \"5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93\": container with ID starting with 5a4946c344886d833bae822bda31a2243e308fc89fb8c4b861720ee1711bac93 not found: ID does not exist"
Sep 17 19:20:17.401199 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:17.401180    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7e891e-predictor-68b7c7dd6d-h57kr"]
Sep 17 19:20:19.238405 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:20:19.238373    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" path="/var/lib/kubelet/pods/1a2009ae-228b-4928-ba95-860cc502c6ff/volumes"
Sep 17 19:21:37.470115 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:37.470085    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j94pq_ebc60a41-a81e-404d-978f-102cc369618b/global-pull-secret-syncer/0.log"
Sep 17 19:21:37.470953 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:37.470935    2577 ???:1] "http: TLS handshake error from 10.0.132.9:35206: EOF"
Sep 17 19:21:37.822096 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:37.822020    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zn5q6_93f04619-04fa-4035-8e10-92990294aa8f/konnectivity-agent/0.log"
Sep 17 19:21:37.855071 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:37.855041    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-128.ec2.internal_86685daa0de7a21d86d5b52a0666ae3d/haproxy/0.log"
Sep 17 19:21:40.923645 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:40.923615    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3267dea6-9e40-494a-aeaa-458e00956620/alertmanager/0.log"
Sep 17 19:21:40.953556 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:40.953534    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3267dea6-9e40-494a-aeaa-458e00956620/config-reloader/0.log"
Sep 17 19:21:40.978775 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:40.978748    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3267dea6-9e40-494a-aeaa-458e00956620/kube-rbac-proxy-web/0.log"
Sep 17 19:21:41.007874 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.007852    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3267dea6-9e40-494a-aeaa-458e00956620/kube-rbac-proxy/0.log"
Sep 17 19:21:41.049053 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.049022    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3267dea6-9e40-494a-aeaa-458e00956620/kube-rbac-proxy-metric/0.log"
Sep 17 19:21:41.075733 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.075704    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3267dea6-9e40-494a-aeaa-458e00956620/prom-label-proxy/0.log"
Sep 17 19:21:41.107235 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.107207    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3267dea6-9e40-494a-aeaa-458e00956620/init-config-reloader/0.log"
Sep 17 19:21:41.200137 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.200040    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-585cx_b3ba0a29-5045-4cf7-9008-998151191fb7/kube-state-metrics/0.log"
Sep 17 19:21:41.228752 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.228726    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-585cx_b3ba0a29-5045-4cf7-9008-998151191fb7/kube-rbac-proxy-main/0.log"
Sep 17 19:21:41.264589 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.264548    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-585cx_b3ba0a29-5045-4cf7-9008-998151191fb7/kube-rbac-proxy-self/0.log"
Sep 17 19:21:41.299471 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.299445    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-65fbc5879c-tlvdb_281b2b78-22d4-43b4-98f1-efbf3e0b140d/metrics-server/0.log"
Sep 17 19:21:41.330588 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.330567    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-7c6x5_88749c0f-07ee-4e7c-97a3-c4afa2b1c78d/monitoring-plugin/0.log"
Sep 17 19:21:41.391957 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.391926    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7rssq_72dd61e0-804b-4b85-abd1-eafbb220388f/node-exporter/0.log"
Sep 17 19:21:41.425868 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.425840    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7rssq_72dd61e0-804b-4b85-abd1-eafbb220388f/kube-rbac-proxy/0.log"
Sep 17 19:21:41.463233 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.463158    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7rssq_72dd61e0-804b-4b85-abd1-eafbb220388f/init-textfile/0.log"
Sep 17 19:21:41.673767 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.673729    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-6wm5x_998c753e-fa09-4b9e-b81b-7deb8aaabfec/kube-rbac-proxy-main/0.log"
Sep 17 19:21:41.703626 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.703581    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-6wm5x_998c753e-fa09-4b9e-b81b-7deb8aaabfec/kube-rbac-proxy-self/0.log"
Sep 17 19:21:41.728170 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.728086    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-6wm5x_998c753e-fa09-4b9e-b81b-7deb8aaabfec/openshift-state-metrics/0.log"
Sep 17 19:21:41.965690 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.965659    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-qntw7_c8414deb-4ab1-4aae-8f38-a48e97d6dfd5/prometheus-operator/0.log"
Sep 17 19:21:41.988835 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:41.988755    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-qntw7_c8414deb-4ab1-4aae-8f38-a48e97d6dfd5/kube-rbac-proxy/0.log"
Sep 17 19:21:42.054784 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:42.054749    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77dccf6945-6j4j7_fcabac82-0ab0-44a4-a296-451115c70543/telemeter-client/0.log"
Sep 17 19:21:42.076609 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:42.076575    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77dccf6945-6j4j7_fcabac82-0ab0-44a4-a296-451115c70543/reload/0.log"
Sep 17 19:21:42.100399 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:42.100374    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77dccf6945-6j4j7_fcabac82-0ab0-44a4-a296-451115c70543/kube-rbac-proxy/0.log"
Sep 17 19:21:42.130855 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:42.130826    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d659bf7cd-r78sr_60025e0d-4f71-4be8-bbfa-8665f8cba082/thanos-query/0.log"
Sep 17 19:21:42.157451 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:42.157424    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d659bf7cd-r78sr_60025e0d-4f71-4be8-bbfa-8665f8cba082/kube-rbac-proxy-web/0.log"
Sep 17 19:21:42.187425 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:42.187397    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d659bf7cd-r78sr_60025e0d-4f71-4be8-bbfa-8665f8cba082/kube-rbac-proxy/0.log"
Sep 17 19:21:42.211465 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:42.211437    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d659bf7cd-r78sr_60025e0d-4f71-4be8-bbfa-8665f8cba082/prom-label-proxy/0.log"
Sep 17 19:21:42.241604 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:42.241530    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d659bf7cd-r78sr_60025e0d-4f71-4be8-bbfa-8665f8cba082/kube-rbac-proxy-rules/0.log"
Sep 17 19:21:42.266760 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:42.266734    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d659bf7cd-r78sr_60025e0d-4f71-4be8-bbfa-8665f8cba082/kube-rbac-proxy-metrics/0.log"
Sep 17 19:21:44.646101 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:44.646026    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56d5947dc-ft2cl_81e08d30-3533-46b2-bdd4-b0a29d47ce1a/console/0.log"
Sep 17 19:21:45.479101 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.479067    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"]
Sep 17 19:21:45.479438 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.479426    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kube-rbac-proxy"
Sep 17 19:21:45.479483 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.479439    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kube-rbac-proxy"
Sep 17 19:21:45.479483 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.479456    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container"
Sep 17 19:21:45.479483 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.479462    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container"
Sep 17 19:21:45.479483 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.479471    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="storage-initializer"
Sep 17 19:21:45.479483 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.479478    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="storage-initializer"
Sep 17 19:21:45.479641 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.479529    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kube-rbac-proxy"
Sep 17 19:21:45.479641 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.479538    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a2009ae-228b-4928-ba95-860cc502c6ff" containerName="kserve-container"
Sep 17 19:21:45.482438 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.482417    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.484970 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.484947    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hmqj2\"/\"openshift-service-ca.crt\""
Sep 17 19:21:45.485094 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.484947    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hmqj2\"/\"kube-root-ca.crt\""
Sep 17 19:21:45.485928 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.485894    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hmqj2\"/\"default-dockercfg-vrw5k\""
Sep 17 19:21:45.490729 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.490700    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"]
Sep 17 19:21:45.558063 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.558023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-sys\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.558249 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.558129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-proc\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.558249 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.558161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-podres\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.558249 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.558201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-lib-modules\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.558249 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.558234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxsmj\" (UniqueName: \"kubernetes.io/projected/85d386e7-8744-4c5e-8aa7-76a7ded45f83-kube-api-access-zxsmj\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.659266 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.659221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-proc\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.659266 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.659274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-podres\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.659694 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.659304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-lib-modules\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.659694 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.659343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxsmj\" (UniqueName: \"kubernetes.io/projected/85d386e7-8744-4c5e-8aa7-76a7ded45f83-kube-api-access-zxsmj\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.659694 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.659347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-proc\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.659694 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.659411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-sys\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.659694 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.659467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-lib-modules\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.659694 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.659465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-podres\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.659694 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.659510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85d386e7-8744-4c5e-8aa7-76a7ded45f83-sys\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.667699 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.667678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxsmj\" (UniqueName: \"kubernetes.io/projected/85d386e7-8744-4c5e-8aa7-76a7ded45f83-kube-api-access-zxsmj\") pod \"perf-node-gather-daemonset-c55w9\" (UID: \"85d386e7-8744-4c5e-8aa7-76a7ded45f83\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:45.793052 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:45.792973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:46.066278 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:46.066214    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"]
Sep 17 19:21:46.068548 ip-10-0-131-128 kubenswrapper[2577]: W0917 19:21:46.068513    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod85d386e7_8744_4c5e_8aa7_76a7ded45f83.slice/crio-b0b3c3c444a0a7ddb562588b32cc8815e29892772bc1874c7112ffb32b134b4e WatchSource:0}: Error finding container b0b3c3c444a0a7ddb562588b32cc8815e29892772bc1874c7112ffb32b134b4e: Status 404 returned error can't find the container with id b0b3c3c444a0a7ddb562588b32cc8815e29892772bc1874c7112ffb32b134b4e
Sep 17 19:21:46.304022 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:46.303992    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tc2zd_be5fcd92-12b9-43df-9e14-a0694aff9fe7/dns/0.log"
Sep 17 19:21:46.327672 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:46.327592    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tc2zd_be5fcd92-12b9-43df-9e14-a0694aff9fe7/kube-rbac-proxy/0.log"
Sep 17 19:21:46.405346 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:46.405323    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pg2qm_a93c6cc9-d649-48f7-94fd-66df7247177c/dns-node-resolver/0.log"
Sep 17 19:21:46.786841 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:46.786800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9" event={"ID":"85d386e7-8744-4c5e-8aa7-76a7ded45f83","Type":"ContainerStarted","Data":"043ab0bc31690253b41c54ac466e3fc8e8b5f72caab1c24bfbdf831271516c51"}
Sep 17 19:21:46.787269 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:46.786862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9" event={"ID":"85d386e7-8744-4c5e-8aa7-76a7ded45f83","Type":"ContainerStarted","Data":"b0b3c3c444a0a7ddb562588b32cc8815e29892772bc1874c7112ffb32b134b4e"}
Sep 17 19:21:46.787269 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:46.786929    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:46.803678 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:46.803633    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9" podStartSLOduration=1.803616245 podStartE2EDuration="1.803616245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:21:46.803364982 +0000 UTC m=+3141.849767903" watchObservedRunningTime="2026-09-17 19:21:46.803616245 +0000 UTC m=+3141.850019179"
Sep 17 19:21:46.888783 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:46.888755    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6cff9d988d-5fl5c_05daf5f5-08e0-4f7c-af04-eeed64c62fb5/registry/0.log"
Sep 17 19:21:46.973168 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:46.973137    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x7n4t_8e8f8317-5d67-48a6-81f8-4856de66a377/node-ca/0.log"
Sep 17 19:21:48.288056 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:48.288023    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rlt2p_0885b1df-0695-4589-8ae5-387f4a25c048/serve-healthcheck-canary/0.log"
Sep 17 19:21:48.821999 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:48.821967    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tlbhh_6f00a5c9-c72a-490c-a33d-2dd121836a18/kube-rbac-proxy/0.log"
Sep 17 19:21:48.846692 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:48.846665    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tlbhh_6f00a5c9-c72a-490c-a33d-2dd121836a18/exporter/0.log"
Sep 17 19:21:48.868831 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:48.868801    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tlbhh_6f00a5c9-c72a-490c-a33d-2dd121836a18/extractor/0.log"
Sep 17 19:21:51.355421 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:51.355387    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-7pz7z_f8c1d308-5cbe-4e05-ae02-92f1721a0ede/manager/0.log"
Sep 17 19:21:51.375266 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:51.375242    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-48tw5_fabfa745-3eaf-4d62-b9f2-895cc8e9fa35/server/0.log"
Sep 17 19:21:51.634141 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:51.634055    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-fkr56_a69ebe86-7eb1-40aa-bf0f-2fa89a8a2013/manager/0.log"
Sep 17 19:21:52.801743 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:52.801713    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-c55w9"
Sep 17 19:21:59.519766 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:59.519740    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xpp64_0833754b-b18a-43ba-bb8d-c5f1c4591a7a/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:21:59.544498 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:59.544465    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xpp64_0833754b-b18a-43ba-bb8d-c5f1c4591a7a/egress-router-binary-copy/0.log"
Sep 17 19:21:59.570183 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:59.570152    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xpp64_0833754b-b18a-43ba-bb8d-c5f1c4591a7a/cni-plugins/0.log"
Sep 17 19:21:59.624160 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:59.624126    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xpp64_0833754b-b18a-43ba-bb8d-c5f1c4591a7a/bond-cni-plugin/0.log"
Sep 17 19:21:59.643174 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:59.643144    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xpp64_0833754b-b18a-43ba-bb8d-c5f1c4591a7a/routeoverride-cni/0.log"
Sep 17 19:21:59.678659 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:59.678633    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xpp64_0833754b-b18a-43ba-bb8d-c5f1c4591a7a/whereabouts-cni-bincopy/0.log"
Sep 17 19:21:59.704195 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:59.704164    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xpp64_0833754b-b18a-43ba-bb8d-c5f1c4591a7a/whereabouts-cni/0.log"
Sep 17 19:21:59.821524 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:59.821444    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-sjv47_e4f408b3-9984-4983-90f5-38a4195a2c2f/kube-multus/0.log"
Sep 17 19:21:59.940404 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:59.940378    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mqx5t_656e881f-1982-4ce2-a57c-299fb552772b/network-metrics-daemon/0.log"
Sep 17 19:21:59.972558 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:21:59.972532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mqx5t_656e881f-1982-4ce2-a57c-299fb552772b/kube-rbac-proxy/0.log"
Sep 17 19:22:00.745168 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:22:00.745139    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5bqq_50caee2e-986c-47b4-9198-ae60bbd7bd90/ovn-controller/0.log"
Sep 17 19:22:00.767193 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:22:00.767169    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5bqq_50caee2e-986c-47b4-9198-ae60bbd7bd90/ovn-acl-logging/0.log"
Sep 17 19:22:00.792413 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:22:00.792388    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5bqq_50caee2e-986c-47b4-9198-ae60bbd7bd90/kube-rbac-proxy-node/0.log"
Sep 17 19:22:00.817456 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:22:00.817428    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5bqq_50caee2e-986c-47b4-9198-ae60bbd7bd90/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:22:00.841223 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:22:00.841183    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5bqq_50caee2e-986c-47b4-9198-ae60bbd7bd90/northd/0.log"
Sep 17 19:22:00.866750 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:22:00.866722    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5bqq_50caee2e-986c-47b4-9198-ae60bbd7bd90/nbdb/0.log"
Sep 17 19:22:00.922794 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:22:00.922770    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5bqq_50caee2e-986c-47b4-9198-ae60bbd7bd90/sbdb/0.log"
Sep 17 19:22:01.000195 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:22:01.000121    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5bqq_50caee2e-986c-47b4-9198-ae60bbd7bd90/ovnkube-controller/0.log"
Sep 17 19:22:02.590452 ip-10-0-131-128 kubenswrapper[2577]: I0917 19:22:02.590426    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-cx897_21c07112-ca6a-4083-9fad-f0b5b5afb0ba/network-check-target-container/0.log"